fix: Ensure the LambdaRuntimeAPI server is up before launching INIT phase

[gregfurman]

Motivation

• Fixes a race condition where the Lambda runtime API server has not yet started up, yet we try to initiate an INIT request.
• Addresses issue in LocalStack community: bug: seems a bug with go lambdas localstack#12680

Changes

• Adds a waitForRuntimeAPI helper function that checks the /ping endpoint of the runtime API server before allowing the INIT functionality to proceed.

[gregfurman]

Context

tl;dr there's a race condition when trying to initialise the RIE before starting the Lambda Runtime API server. The solution is to block and poll the runtime server until we're certain it's up.

Problem

When building a Sandbox, the RIE automatically starts the runtime API server in the background.

1. We create the sandbox via SandboxBuilder.Create() method:

lambda-runtime-init/cmd/localstack/main.go

Lines 214 to 215 in 9be92e0

	// initialize all flows and start runtime API
	sandboxContext, internalStateFn := sandbox.Create()

2. This then triggers the rapid.Server to be initialised and started (allowing the RIE to accept /invocation/next calls among others):

lambda-runtime-init/lambda/rapidcore/sandbox_builder.go

Line 175 in 9be92e0

rapidCtx, internalStateFn, runtimeAPIAddr := rapid.Start(ctx, b.sandbox)

3. After initialising, Start actually launches the runtime API server in a seperate goroutine and returns. This, however, does not guarantuee that the server has actually started.

lambda-runtime-init/lambda/rapid/sandbox.go

Line 106 in 9be92e0

go startRuntimeAPI(ctx, execCtx)

4. Our Init call then fails since there's a chance the runtime server has not yet started:

lambda-runtime-init/cmd/localstack/main.go

Line 231 in 9be92e0

InitHandler(sandbox.LambdaInvokeAPI(), GetEnvOrDie("AWS_LAMBDA_FUNCTION_VERSION"), int64(invokeTimeoutSeconds), bootstrap, lsOpts.AccountId) // TODO: replace this with a custom init

Solution

Instead of relying on the good graces of the scheduler to start the LambdaRuntimeAPI server before trying to initialise, we block after the SandboxBuilder.Create() until the/ping handler (Hyrum's law proven correct once again) resolves.

lambda-runtime-init/lambda/rapi/router.go

Lines 29 to 32 in 9be92e0

	// To respect Hyrum's Law, keeping /ping API even though
	// we no longer use it ourselves.
	// http://www.hyrumslaw.com/
	router.Get("/ping", handler.NewPingHandler().ServeHTTP)

[dfangl]

Good work! I think this should resolve the issue, however I am a bit concerned about the possible increase in startup time. Let's discuss the timeout before merging!

[dfangl]

Did you per chance test how many - on average - retries we get here, and how this affects the average startup times? Does it make sense to reduce this to 10 - 50ms, for example? More pings, but less delay? The startup time should be significantly less than 10ms ideally anyway, right?

[gregfurman]

@dfangl Fair points. I saw these in the order of milliseconds based on my local runs (keeping in mind that I'm on ARM64 and don't have issues with this).

{"file":"lambda-runtime-init/lambda/rapi/server.go:108","func":"go.amzn.com/lambda/rapi.(*Server).Listen","level":"debug","msg":"Runtime API Server listening on 127.0.0.1:9001","time":"2025-10-17T19:35:58+02:00"}
{"file":"lambda-runtime-init/lambda/rapi/middleware/middleware.go:76","func":"go.amzn.com/lambda/rapi/middleware.AccessLogMiddleware.func1.1","level":"debug","msg":"API request - GET /2018-06-01/ping, Headers:map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]","time":"2025-10-17T19:35:59+02:00"}
{"file":"lambda-runtime-init/lambda/rapi/middleware/middleware.go:76","func":"go.amzn.com/lambda/rapi/middleware.AccessLogMiddleware.func1.1","level":"debug","msg":"API request - GET /2018-06-01/ping, Headers:map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]","time":"2025-10-17T19:35:59+02:00"}
{"file":"ambda-runtime-init/cmd/localstack/main.go:245","func":"main.main","level":"debug","msg":"Starting runtime init.","time":"2025-10-17T19:36:04+02:00"}

In anycase, these checks are probably more conservative than what is necessary so increasing frequency and decreasing timeout duration seems logical.

Otherwise, if we're trying to make this as fast as possible, so as to not delay startup times, we can also do a single check to see if the port is open (similar to LocalStack's is_port_open()) with some timeout of 5 seconds (or something equivalently short).

[dfangl]

I'm fine either way - I just think 500ms for delay is too much, especially since the first one will for quite some systems fail.

[gregfurman]

OK changed to 50ms! Happy to 🚢 ?

[dfangl]

LGTM! Thanks for increasing the poll rate!