Some commands can hang for very long periods before failing; please consider adding timeout

[smithfarm]

Certain commands, like "cephadm bootstrap" and podman pull can conceivably get stuck and hang for long periods of time, or even "forever".

It's worth considering to precede these commands with timeout.

This is not just for interactive users sitting at a console - the primary purpose is to ensure that CI jobs fail within a reasonable amount of time when commands get stuck.

[sebastian-philipp]

how long is too long? what about deployments with a slow download connection?

[smithfarm]

how long is too long? what about deployments with a slow download connection?

The timeout could be configurable, and default to a large number, like one hour.

[sebastian-philipp]

Maybe we can improve this, if we separate the pull from calling bootstrap?

[bk201]

Maybe we can improve this, if we separate the pull from calling bootstrap?

We already did that. Images are pulled before bootstrap.

[smithfarm]

Normally the cephadm bootstrap step completes within 5 minutes for me. But occasionally it just hangs. It just happened again today.

[sebastian-philipp]

we need to fix the underlying cause of this. Please have a look at where cephadm actually hangs.

[smithfarm]

we need to fix the underlying cause of this. Please have a look at where cephadm actually hangs.

Indeed. Yesterday I found out that cephadm bootstrap logs to /var/log/ceph/cephadm.log :-)

[sebastian-philipp]

can we close this?

[ricardoasmarques]

I was not able to reproduce this issue.

One situation where ceph-salt commands will hang is when ceph cluster is stoped, and that situation is tracked on a separate issue: #409

If there's any other situation where a timeout is still mission feel free to reopen or submit a separate issue for that.