Repository navigation
Time out deploy commands after 10 minutes - #19
Merged
Merged
Conversation
A deploy command could hang forever, such as on a prompt to accept a certificate, holding its thread with the job never finished. Wait at most 10 minutes for the command, then close its channel and record a SocketTimeoutException with the output so far. The async deploy also no longer runs in a transaction: Wildfly's default transaction timeout is 300 seconds, so a command running longer than that left a rolled-back transaction and the job was never saved. The job is saved by deployJobFacade.edit in its own transaction. Fixes #1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slominskir
approved these changes
Oct 1, 2026
slominskir
left a comment
Member
There was a problem hiding this comment.
Some C++ app deploys build from source, and that can take a very long time, especially if Clang Tidy static analysis is repeated (generally done in CI). We'll likely need to revisit this hard-coded 10 minute timeout, but good enough for now.
This was referenced Oct 1, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1.
What changed
SSHFacadewaits at most 10 minutes (commandTimeout) for the deploy command. After that it closes the channel and the job records aSocketTimeoutExceptionas its stack trace, with no exit code, and the stdout and stderr received so far.asyncExecuteRemoteCommandis now@TransactionAttribute(NOT_SUPPORTED). Before, it ran in a container transaction for the whole command. Thejeffersonlab/wildfly:3.0.1image keeps Wildfly's default transaction timeout of 300 seconds, so a command running longer than that would leave the transaction rolled back, and saving the job at the end would fail.deployJobFacade.edit(job)now saves the job in its own transaction.The 10 minutes is a hard-coded constant, like the other SSH timeouts. If some deploys need longer, it could become an environment variable or a column on
APP_ENV.No deployment steps.
What happens on the remote host
The sshd library has no client API to signal the remote process, so the timeout only closes the channel:
sleep, keeps running on the remote host until it ends on its own.Checks
gradle spotlessApply buildpasses (JDK 21, in thegradle:9-jdk21container).executeRemoteCommand(through reflection, withcommandTimeoutset to 3 seconds) against the project'sDockerfile-sshdcontainer:deploy_command(version1.0.0appended)touch /tmp/hello && echo1.0.0sh -c 'echo out; echo err >&2; exit 3' #out/errsh -c 'echo partial; sleep 30; echo done' #partialSocketTimeoutExceptionsh -c 'echo asking; read -r x; echo got:$x' #askingSocketTimeoutExceptionAfter the run, the
sleepcommand was still running in the sshd container; thereadcommand had exited.Not checked: the transaction change and the
/logpage in the full Compose stack. Port 8081 on the VM is taken by another project's Keycloak.🤖 Generated with Claude Code