Get resource is stuck #9398

kristianstanchev · 2025-12-15T15:01:35Z

kristianstanchev
Dec 15, 2025

Summary

When trying to get resource (in this case type is 'time') Concourse is stuck in the following step:

When we aborted the original build that was getting the resource all other pending 'gets' succeeded.
There were no worker/web logs suggesting any possible reason or issues.

Steps to Reproduce

Issue reproduced on multiple AWS environments, but we still don't have any concrete steps how to reproduce the issue.

Expected Results

Get step used to pass always in a matter of seconds.

Actual Results

Get step is stuck until timeout or abortion of the stuck get.

Web Node(s) configuration

We have multiple web nodes configured with:

CONCOURSE_ENABLE_GLOBAL_RESOURCES=true

which I suspect might be relevant for the behavior

Worker(s) configuration

We have multiple workers and recently (a couple of days before the issue appeared) switched the runtime to containerd

Concourse Version

7.14.3

Browser (if applicable)

N/A

Did this use to work?

No issues with previous versions and guardian runtime

Answered by Kump3r

Apr 3, 2026

Converting to issue #9511

View full answer

taylorsilva · 2025-12-15T16:25:49Z

taylorsilva
Dec 15, 2025
Maintainer

Moved this over to a discussion and then we can make an issue if we figure out concrete steps to reproduce.

I think you've provided enough info here. To summarize what I'm seeing so far:

Resource used is the time resource
Global resources is enabled
On 7.14.3
containerd runtime is being used

Some assumptions I'll make when trying to reproduce this:

Multiple pipelines using the same resource are required. Can probably set the same pipeline config multiple times, each with different names
mhmmm can't think of any others right now so I guess that's the end of this list 😅

@Kump3r This sounds similar to the issue you were DM'ing me about, doesn't it?

10 replies

Kump3r Dec 16, 2025
Collaborator

Debug logs for the whole flow:

{"timestamp":"2025-12-16T13:42:28.448324595Z","level":"debug","source":"atc","message":"atc.tracker-imb.received-in-memory-build","data":{"build":"check","pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.449260220Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.lock.acquire.acquired","data":{"build":"check","id":[8,-1771735191],"pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.1.1","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.449338928Z","level":"info","source":"atc","message":"atc.tracker-imb.run.running","data":{"build":"check","pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.450492053Z","level":"info","source":"atc","message":"atc.tracker-imb.run.check-step.initializing","data":{"build":"check","pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.2","step-name":"1-minute","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.456829803Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.acquire.acquired","data":{"build":"check","id":[0,1],"pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.3","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.457213553Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.check-step.after-wait-to-run","data":{"build":"check","pipeline":"test","pre_build_id":4,"resource":"1-minute","run":true,"scope":4,"session":"23.4.2","step-name":"1-minute","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.468941553Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.check-step.found-created-container-in-db","data":{"build":39,"container":"9eab9b4f-8236-4ce5-8f11-481782b7537a","pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.2","step-name":"1-minute","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.474738178Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.check-step.starting","data":{"build":39,"pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.2","step-name":"1-minute","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.542634053Z","level":"info","source":"atc","message":"atc.tracker-imb.run.check-step.finished","data":{"build":39,"pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.2","step-name":"1-minute","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.542776595Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.release.released","data":{"build":"check","id":[0,1],"pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.4","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.543687303Z","level":"info","source":"atc","message":"atc.tracker-imb.run.finish.succeeded","data":{"build":"check","build_id":39,"pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.5","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.543814553Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.lock.release.released","data":{"build":"check","id":[8,-1771735191],"pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.1.2","team":"main"}}

I will attach what was the last info, we received. Edit: atc.tracker-imb.run.check-step.initializing
This narrows it down to one of the following processes that hanged:

{"timestamp":"2025-12-16T13:42:28.456829803Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.acquire.acquired","data":{"build":"check","id":[0,1],"pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.3","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.457213553Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.check-step.after-wait-to-run","data":{"build":"check","pipeline":"test","pre_build_id":4,"resource":"1-minute","run":true,"scope":4,"session":"23.4.2","step-name":"1-minute","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.468941553Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.check-step.found-created-container-in-db","data":{"build":39,"container":"9eab9b4f-8236-4ce5-8f11-481782b7537a","pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.2","step-name":"1-minute","team":"main"}}
{"timestamp":"2025-12-16T13:42:28.474738178Z","level":"debug","source":"atc","message":"atc.tracker-imb.run.check-step.starting","data":{"build":39,"pipeline":"test","pre_build_id":4,"resource":"1-minute","session":"23.4.2","step-name":"1-minute","team":"main"}}

taylorsilva Dec 17, 2025
Maintainer

I think you might be getting a bit off track by looking at the in-memory build stuff. That only applies to resource checks. OP's issue appears to be happening with the get step.

Not completely ignoring what you've raised though. It does sound like there is something off with the in-memory build code.

Kump3r Dec 17, 2025
Collaborator

But doesn’t get also do a nested check, or is that old news. Cause the logs suggest that that is the last emitted log for that resource in our case. Maybe for OP it is different. Just by filtering the Kibana, after the failed log it started succeding again and gets haven’t hit the issue anymore. Just making observations, anyway thanks for the direction if anyone has idea what to test for reproducing I will be more than happy to. Last I killed the containerd process on one of the workers but that seem to had a totally different effect with logs and metrics an everything.

Kump3r Dec 17, 2025
Collaborator

TLDR: Maybe the check blocks the get is what I mean 😅

Kump3r Dec 17, 2025
Collaborator

Is there anything regarding the runtime that is persisted in the web, or db? That also might be worth checking. Maybe it got conflicted. As for our use case it happened a couple of days after migrating to containerd on those environments and OP also mentioned they use containerd and didn’t see similar issues with guardian which makes me think they also might have migrated at some point

Kump3r · 2025-12-16T14:01:24Z

Kump3r
Dec 16, 2025
Collaborator

well sound similar to #8639 as far as I can tell

0 replies

Kump3r · 2026-01-06T08:36:31Z

Kump3r
Jan 6, 2026
Collaborator

From what I was able to deduce purely codewise, the fact to the matter is that the time-resource has a problem with the ttrpc, the check of the time resource how I understand it emits a single request and there is a case when this leaks is missed and the ttrpc is stuck waiting for the event, emitting logs like:

time="2026-01-04T22:27:41.728665994Z" level=error msg="ttrpc: received message on inactive stream" stream=1

and eventually:

time="2026-01-04T22:27:41.171253938Z" level=error msg="post event" error="context deadline exceeded"

Not sure if here the idea would be to adjust the Run/Wait of our containerd implementation to something similar to #8639, or actually trying to make the time resource more robust, to ensure somehow the version is "received". Still trying to have a full-proof way to reproduce, but our clusters that have a lot of "check-every" time resources seem to hit this from time to time.
I also saw a check_timeout configuration that might help mitigate this for the check at least, but I assume it would not work for the get which OP described.

0 replies

Kump3r · 2026-01-06T10:38:56Z

Kump3r
Jan 6, 2026
Collaborator

Also a check that should create a new version every 5-min and has a check_every 1m, has a gap/leak for over an hour:

I suspect that after the hour passed the default check_timeout of 1h. hit and the next check succeeded.

0 replies

Kump3r · 2026-01-06T13:53:51Z

Kump3r
Jan 6, 2026
Collaborator

I think I was able to confirm a leak at least, by:

Attaching to the webs of an impacted instance
Checking the lidar metrics with curl http://localhost:9090 | grep concourse_lidar
Doing the math STARTED − (SUCCESS + ERROR) should eventauly equal 0
That was true for all web hosts, except for a single one which comes out to 1
Example:

web/...:~$ curl http://localhost:9090 | grep concourse_lidar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3922    0  3922    0     0  1001k      0 --:--:-- --:--:-- --:--:--  957k# HELP concourse_lidar_checks_enqueued_total Total number of checks enqueued
# TYPE concourse_lidar_checks_enqueued_total counter
concourse_lidar_checks_enqueued_total{bosh_deployment="concourse",bosh_job="web"} 16616
# HELP concourse_lidar_checks_finished_total Total number of checks finished.
# TYPE concourse_lidar_checks_finished_total counter
concourse_lidar_checks_finished_total{bosh_deployment="concourse",bosh_job="web",status="error"} 1292
concourse_lidar_checks_finished_total{bosh_deployment="concourse",bosh_job="web",status="success"} 12702
# HELP concourse_lidar_checks_started_total Total number of checks started. With global resource enabled, a check build may not really run a check, thus total checks started should be less than total check builds started.
# TYPE concourse_lidar_checks_started_total counter
concourse_lidar_checks_started_total{bosh_deployment="concourse",bosh_job="web"} 13995
100 1113k    0 1113k    0     0  85.9M      0 --:--:-- --:--:-- --:--:-- 83.6M

Sadly there is no metric for concourse_lidar_checks_finished_total{bosh_deployment="concourse",bosh_job="web",status="timeout"} X to confirm it 100%, but I assume we reached the 1h default timeout here, and hence we do not see success/error metric resulting in the diff.

0 replies

Kump3r · 2026-01-15T10:18:34Z

Kump3r
Jan 15, 2026
Collaborator

I have seen also a stucked task, what I was able to pull as info from our side:

Environment

Concourse worker: containerd backend
containerd: v2.1.5
runc: 1.3.3 (spec 1.2.1, libseccomp 2.5.6)
Kernel: healthy (no blocked tasks, no D-state processes)

Symptoms

Pipeline step hangs until timeouts are reached. Intermittent usually right after container start

Garden logs show attach failure immediately after container creation (this is normal as attach is always tried first in the concourse implementation):
```
worker.garden.garden-server.attach.failed
error="load proc: no running process found: process does not exist task: not found"
```

containerd reports the task as RUNNING::

ctr -n concourse tasks list
c35e03c8-09f5-48f1-9575-bbcbc0eea82e  <pid>  RUNNING

Process state:

The container task PID corresponds to gdn-init:

ps -o pid,ppid,state,wchan,cmd -p <pid>

PID      PPID  S  WCHAN   CMD
<id>     <id>  S  do_sys  /tmp/gdn-init

Process tree:

systemd
 └─ containerd-shim
     └─ gdn-init

Observations:

gdn-init is alive
It has no children
The user process (/bin/bash) was never exec’d
State is S (interruptible sleep), not D
Blocked in do_sys (syscall wait)
This indicates a userspace handshake deadlock, not kernel I/O.

Notably NOT the cause

Explicitly ruled out:

Disk / overlayfs / NFS issues
Kernel blocked tasks
Uninterruptible sleep
Missing containerd task
Old runtime versions (this reproduces on containerd v2.1.5 + runc 1.3.3)

Trigger characteristics

Occurs during garden-server.run
TTY requested (TTY.window_size present)
High container churn seems to increase the likelihood

1 reply

Kump3r Jan 15, 2026
Collaborator

maybe we can try context.with timeout here:

concourse/worker/runtime/container.go

Lines 99 to 101 in e81401c

    
           ctx := context.Background() 
        
           containerSpec, err := c.container.Spec(ctx)

instead maybe do:

	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()	containerSpec, err := c.container.Spec(ctx)

still not sure how to reproduce it though

Kump3r · 2026-01-16T08:20:18Z

Kump3r
Jan 16, 2026
Collaborator

Logs during reproduce compared to another check before it of the same resource

0 replies

Kump3r · 2026-01-21T09:41:48Z

Kump3r
Jan 21, 2026
Collaborator

Potentially when you have a lot of resource versions, that is more reproducible, particularly in our case around 20 000 of that resource.

0 replies

Kump3r · 2026-04-03T13:41:05Z

Kump3r
Apr 3, 2026
Collaborator

Converting to issue #9511

0 replies

Concourse

Get resource is stuck #9398

Uh oh!

kristianstanchev Dec 15, 2025

Summary

Steps to Reproduce

Expected Results

Actual Results

Web Node(s) configuration

Worker(s) configuration

Concourse Version

Browser (if applicable)

Did this use to work?

Replies: 9 comments · 11 replies

Uh oh!

Uh oh!

taylorsilva Dec 15, 2025 Maintainer

Uh oh!

Uh oh!

Kump3r Dec 16, 2025 Collaborator

Uh oh!

taylorsilva Dec 17, 2025 Maintainer

Uh oh!

Kump3r Dec 17, 2025 Collaborator

Uh oh!

Kump3r Dec 17, 2025 Collaborator

Uh oh!

Kump3r Dec 17, 2025 Collaborator

Uh oh!

Kump3r Dec 16, 2025 Collaborator

Uh oh!

Kump3r Jan 6, 2026 Collaborator

Uh oh!

Kump3r Jan 6, 2026 Collaborator

Uh oh!

Kump3r Jan 6, 2026 Collaborator

Uh oh!

Kump3r Jan 15, 2026 Collaborator

Environment

Symptoms

Process state:

Observations:

Notably NOT the cause

Explicitly ruled out:

Trigger characteristics

Uh oh!

Kump3r Jan 15, 2026 Collaborator

Uh oh!

Kump3r Jan 16, 2026 Collaborator

Uh oh!

Kump3r Jan 21, 2026 Collaborator

Uh oh!

Uh oh!

Kump3r Apr 3, 2026 Collaborator

kristianstanchev
Dec 15, 2025

Replies: 9 comments 11 replies

taylorsilva
Dec 15, 2025
Maintainer

Kump3r Dec 16, 2025
Collaborator

taylorsilva Dec 17, 2025
Maintainer

Kump3r Dec 17, 2025
Collaborator

Kump3r Dec 17, 2025
Collaborator

Kump3r Dec 17, 2025
Collaborator

Kump3r
Dec 16, 2025
Collaborator

Kump3r
Jan 6, 2026
Collaborator

Kump3r
Jan 6, 2026
Collaborator

Kump3r
Jan 6, 2026
Collaborator

Kump3r
Jan 15, 2026
Collaborator

Kump3r Jan 15, 2026
Collaborator

Kump3r
Jan 16, 2026
Collaborator

Kump3r
Jan 21, 2026
Collaborator

Kump3r
Apr 3, 2026
Collaborator