feat(benchmark): pool benchmark harness with standalone mock server and per-API QPS tracking by Pangjiping · Pull Request #1518 · opensandbox-group/OpenSandbox

Pangjiping · 2026-08-14T07:24:25Z

Summary

Adds a reproducible, cross-SDK benchmark harness for the sandbox client pool under tests/benchmark/:

Standalone Go mock server (stdlib only) implementing the lifecycle + execd API surface per specs/, with per-route response-time control (uniform/fixed/lognormal), fault injection (create failures, execd failures, partial sandbox poisoning), server-side TTL expiry, and exact per-second per-API QPS tracking with full time series.
Kotlin/JVM driver running 10 reproducible scenarios against the mock: cold-start, warm-latency, steady-state (with rate limiting via --acquire-rate-per-min), replenish-lag, failure-injection, stale-idle, idle-expiry, resize, shutdown-race, store-outage. Reports acquire latency percentiles, success rate + failure classification, pool health (idle trajectory, degraded/backoff, in-flight), replenish throughput, client threads/heap/GC, and per-scenario server QPS.
run.sh orchestrates mock build/start + driver run; every artifact of a run (reports, per-second timeseries CSVs, client probe CSVs, mock config, args, logs) lands in one results/run-<ts>/ directory.
The Kotlin SDK is built from source via a Gradle composite build (includeBuild), so the harness always runs the checked-out SDK.

SDK companion

PoolWarmupDiagnostics (diagnostic-only, zero-config, resettable) was added to the Kotlin SDK to instrument the warmup pipeline (queue-wait, create duration, commit, tick cadence, in-flight peak, failure reasons). It powered the diagnosis in #1514. It is additive and non-breaking; reviewers may prefer it dropped or formalized separately.

Evidence produced

pool: killed idle slot is not refilled until the next periodic reconcile (up to reconcileInterval) #1513: killed idle slots are not refilled until the next periodic reconcile (replenish-lag measured up to reconcileInterval).
pool: warmupConcurrency beyond ~200 does not increase fill throughput (effective in-flight concurrency caps at ~200-250) #1514: warmupConcurrency scaling — per-sandbox fresh-TCP-connection churn at high concurrency causes connection resets and retry amplification; fixed independently in PR feat(pool): default shared connection pool sized by warmupConcurrency #1517.

Verification

go build/vet on the mock server; ./gradlew build on the driver; full 7+3 scenario runs and a 30-minute steady-state run (2000/min, maxIdle=13815) exercised end-to-end.
Usage and all knobs documented in tests/benchmark/README.md.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e533355c29

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9c3ecc7d36

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-14T07:45:51Z

+        repeat(cfg.steadyWorkers) {
+            threads.submit {
+                while (System.nanoTime() < deadline) {
+                    pacer.waitForSlot()


Recheck the deadline after pacing each acquire

When --acquire-rate-per-min is low relative to --steady-workers and --steady-duration-s, workers can enter the loop before the deadline, reserve slots far in the future in RatePacer, and then still perform pool.acquire() after the scenario duration has expired. For example, 16 workers at 1 acquire/minute can reserve slots out to ~15 minutes during a 60-second run; awaitTermination(duration+300s) can return while non-daemon worker tasks are still sleeping/acquiring, so the benchmark duration and metrics are inflated and the process may keep running long after the configured scenario window.

Useful? React with 👍 / 👎.

Fixed in 8c25c68: workers recheck the scenario deadline after RatePacer sleeps and break before acquiring past the run window.

chatgpt-codex-connector · 2026-08-14T07:45:51Z

+# 4. run the driver (Kotlin SDK is built from source via composite build,
+#    see kotlin/settings.gradle.kts)
+echo "== running benchmark driver =="
+DRIVER_ARGS+=("--report-dir" "${RUN_DIR}")


Pass overridden mock address to the driver

When users set the documented LIFECYCLE_ADDR to run the mock on a non-default port or host, the script health-checks that address but never forwards it to the Kotlin driver, so Cli.parse falls back to http://127.0.0.1:18080 and the run fails unless the user redundantly supplies --mock-base-url. Add the derived http://${LIFECYCLE_ADDR} to DRIVER_ARGS (unless already provided) alongside the report dir so the orchestration works with the advertised environment override.

Useful? React with 👍 / 👎.

Fixed in 8c25c68: run.sh now forwards the effective LIFECYCLE_ADDR to the driver as --mock-base-url unless the user pinned it.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f367f7b033

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-14T08:35:46Z

+	token := r.Header.Get("X-EXECD-ACCESS-TOKEN")
+	id := strings.TrimPrefix(token, "mock-token-")
+	if token != "" && token == execdToken(id) {


Reject execd requests without a valid endpoint token

When an execd request is missing X-EXECD-ACCESS-TOKEN or sends a malformed token, this condition is skipped and the handler falls through to a 200 response. That means any SDK/client path that drops endpoint headers will appear ready immediately, bypassing the mock's Pending/expired/poisoned checks and invalidating readiness, stale-idle, and cross-SDK header-propagation benchmark results; reject missing or invalid tokens instead of treating them as unauthenticated success.

Useful? React with 👍 / 👎.

Fixed in 8c25c68: execd requests without a valid per-sandbox endpoint token are rejected with 401 instead of answered 200.

chatgpt-codex-connector · 2026-08-14T08:35:46Z

+            } catch (t: Throwable) {
+                System.err.println("scenario $name failed: $t")
+                mapOf("error" to (t.message ?: t.toString()))


Clean up failed scenarios before continuing

When a scenario throws after starting its pool, this catch records the error and then proceeds to collect stats and run the remaining scenarios. For example, an unguarded acquire in replenish-lag can throw before its pool.shutdown, leaving reconcile/warmup threads and live mock sandboxes active so later QPS/alive metrics are contaminated and the JVM may keep running; either abort here or ensure every scenario shuts its pool down in finally before returning an error section.

Useful? React with 👍 / 👎.

Fixed in 8c25c68: every scenario wraps its pool lifecycle in try/finally (return try { ... } finally { pool.shutdown(...) }), so a failed scenario cannot leak threads or mock sandboxes.

…er-API QPS tracking Standalone Go mock server (lifecycle + execd API per specs/) with configurable per-route response time (uniform/fixed/lognormal), fault injection (create failure, execd failure, poisoned sandboxes), server-side TTL expiry, and exact per-second per-API QPS tracking with full time series. Kotlin/JVM driver runs 7 reproducible scenarios against the mock (cold-start, warm-latency, steady-state, replenish-lag, failure-injection, stale-idle, idle-expiry) and writes JSON + Markdown reports including per-scenario server-side QPS. run.sh orchestrates SDK publish, mock start, and driver run; the mock is reusable from any SDK via ConnectionConfig.

…e build Replace the mavenLocal-published artifact dependency with includeBuild on sdks/sandbox/kotlin so the benchmark always runs the checked-out SDK code. Drops the publishToMavenLocal step from run.sh and the -PsandboxVersion injection.

…ainingTtl, primaryLockTtl, degradedThreshold) Map a production large-pool / high-frequency profile 1:1 onto driver options; document the example profile and scale caveats in README.

…ofile Keeps the high-frequency acquire case realistic without the 1-5s readiness probe capping the sustainable acquire rate.

…/replenish metrics PoolProbe samples JVM threads, heap/GC, and pool health every 500ms during warm-latency and steady-state. Scenarios now report successRate, replenishRatePerSec/killRatePerSec, directCreateRatio, and the client probe block; steady-state idle stats come from the probe.

run.sh creates results/run-<ts>/ holding the mock log, mock config, driver args, reports, per-scenario server timeseries CSVs (alive + per-API QPS) and client probe CSVs (threads/heap/idle/inFlight), plus a raw end-of-run mock stats snapshot. Adds failure classification (readyTimeout/poolNotRunning/ storeUnavailable/...), resize, shutdown-race, store-outage scenarios, partial poisoning, and mock alive gauge (max + per-second series).

…ation awaitTermination(15min) silently truncated 30-min runs to ~15min. Wait for duration+5min and report the actual loader duration so achieved rate is computed against real runtime.

…se experiments PoolWarmupDiagnostics records warmup queue-wait, createOneSandbox duration, commit duration, reconcile-tick cadence, in-flight warmup peak, and create failure reasons. Cold-start scenario reports them; --shared-connection-pool flag added to test connection-reuse hypotheses. Diagnosis: at wc=1000 the submission chain fully saturates (in-flight peak 1000, queue wait ~20ms); fill degradation is caused by per-sandbox fresh-TCP connection churn producing intermittent Connection reset failures against the mock listener (accept backlog), amplifying attempts via retries.

…on-pool-size) Replace the boolean flag with a configurable OkHttp ConnectionPool idle size. Evidence: at wc=1000, pool size 0/100/200/500 -> fill 19.9s/9.1s/7.1s/4.3s, connection-reset failures 8024/2696/1341/0, attempt amplification 5x/2.3x/ 1.7x/1x. A properly sized shared pool eliminates the TCP churn entirely.

…pool guidance Adds --shared-connection-pool-size to the driver options table and a 'Warmup throughput and connection reuse' section describing the connection-reset/retry-amplification finding at high warmupConcurrency, the pool-size sweep evidence, and configuration guidance for benchmark and production (including the upcoming SDK default from opensandbox-group#1517).

…ent pool guide Move the problem description, sweep evidence, and configuration guidance (shared ConnectionPool via ConnectionConfig, sizing rule of thumb, upcoming SDK default) to docs/guides/client-pool.md. The benchmark README keeps only a pointer plus the --shared-connection-pool-size repro flag.

…ized) --shared-connection-pool-size now defaults to auto = max(warmupConcurrency, 200) instead of 0, matching the guidance documented in docs/guides/client-pool.md. Explicit 0 disables sharing (reproduces the connection-reset pathology); N sweeps pool sizes. Report shows the resolved mode ('auto'/0/N).

Replace the auto-derived pool size with a plain fixed default (500 idle slots, 5min keep-alive) so the benchmark follows the documented guidance without any derivation logic. 0 still disables sharing to reproduce the connection-reset pathology; N sweeps sizes.

… scenario Loaders race the pool fill instead of waiting for idle; fixes the CLI parser for valueless boolean flags (previously swallowed the next --key as its value).

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e28aaaf833

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-17T05:06:18Z

+func (m *MockServer) handleCreate(w http.ResponseWriter, r *http.Request) {
+	start := time.Now()
+	defer m.recordQps("lifecycle.create", start)
+	_, _ = io.Copy(io.Discard, r.Body)


Honor the requested timeout when creating sandboxes

The Kotlin pool sends its configured idleTimeout in CreateSandboxRequest.timeout, but this handler discards the request body and always calculates expiry from defaultTtlSeconds. Consequently --idle-timeout-s does not control the mock's server-side TTL, and the default idle-expiry scenario's 20-second sandboxes remain server-alive for 3600 seconds, so it measures local pool reaping rather than the advertised server-expiry behavior. Parse the request timeout and use it when present.

AGENTS.md reference: AGENTS.md:L43-L44

Useful? React with 👍 / 👎.

Fixed in 8c25c68: handleCreate now parses CreateSandboxRequest.timeout and applies it to the server-side TTL (verified: timeout=30 -> expiresAt now+30s); idle-expiry now measures real server expiry.

chatgpt-codex-connector · 2026-08-17T05:06:18Z

+		m.mu.RLock()
+		sb := m.sandboxes[id]
+		m.mu.RUnlock()
+		if sb == nil || !sb.running(time.Now()) {


Keep sandbox fields protected while reading them

During concurrent boot, renewal, deletion, or poisoning, the handler releases m.mu before reading State, ExpiresAt, and Poisoned through sb. Those fields are written under m.mu, so normal readiness traffic races with the mock's mutation goroutines and can observe inconsistent state; the same pattern occurs in lifecycle get/renew/endpoint handlers. Hold the read lock through the checks or copy the sandbox value while locked.

Useful? React with 👍 / 👎.

Fixed in 47b6dd4: the execd handler now reads and judges State/ExpiresAt/Poisoned while holding m.mu.

chatgpt-codex-connector · 2026-08-17T05:06:18Z

+                latency.recordFailure(classifyFailure(t))
+            }
+        }
+        val stats = mock.stats()


Collect stale-idle stats after the refill completes

When stale candidates leave asynchronous warmups still creating, this snapshot is taken before waitForIdle waits for recovery. Any creations completing during that wait are therefore absent from serverCreatedDelta, and serverAliveAfter describes the pre-refill state, making the scenario unable to validate the fresh refill it claims to report. Take the stats snapshot after the refill wait.

Useful? React with 👍 / 👎.

Fixed in 47b6dd4: stale-idle snapshots server stats after the refill wait, so creations completing during recovery are included.

chatgpt-codex-connector · 2026-08-17T05:06:18Z

+            if (slot == 0L) {
+                if (nextSlot.compareAndSet(0L, System.nanoTime())) {
+                    slot = System.nanoTime()


Reserve the next interval after the first pacing slot

With pacing enabled, the first caller stores the current time in nextSlot and runs immediately, but the second caller claims that same stored timestamp and also runs immediately; only the third caller waits one interval. This doubles the initial allowance, so short or low-rate runs can substantially exceed the requested rate—for example, a one-minute run at one acquire per minute starts two acquires instead of one. Initialize nextSlot to the first caller's next interval rather than its current slot.

Useful? React with 👍 / 👎.

Fixed in 47b6dd4: the first caller runs immediately but reserves the next slot one interval out, so the second caller waits one interval instead of doubling the initial allowance.

chatgpt-codex-connector · 2026-08-17T05:06:18Z

+	if cur := s.maxCreateLatency.Load(); ms > cur {
+		s.maxCreateLatency.CompareAndSwap(cur, ms)
+	}


Retry the atomic maximum update after contention

Under concurrent creates, a larger latency can lose its single CompareAndSwap to a smaller concurrent update and then never retry. For example, samples of 100 ms and 200 ms can leave maxCreateLatencyMs at 100 ms, corrupting the server latency summary precisely in high-concurrency benchmark runs. Use a CAS loop or derive the maximum from the already locked sample slice.

Useful? React with 👍 / 👎.

Fixed in 47b6dd4: max-latency update now loops the CAS so a larger concurrent sample cannot be lost.

… robustness - warm-latency hitRatio: subtract replenish creates (one per kill) before computing misses, so background refills no longer depress the ratio - steady-state: recheck the scenario deadline after RatePacer sleeps so workers cannot issue acquires past the run window - run.sh: forward the effective LIFECYCLE_ADDR to the driver as --mock-base-url unless the user pinned it - mock execd: reject requests without a valid per-sandbox endpoint token (401) instead of answering 200, so missing headers cannot fake readiness - mock create: honor CreateSandboxRequest.timeout for the server-side TTL (idle-expiry now measures real server expiry) - scenarios: wrap every pool lifecycle in try/finally so a failed scenario cannot leak reconcile/warmup threads or mock sandboxes

- RatePacer: reserve the next slot one interval after the first caller so the second caller does not also run immediately (doubled initial allowance) - mock execd: judge sandbox state while holding the lock (boot/renew/delete/ poison write those fields under m.mu) - mock stats: retry the max-latency CAS so a larger concurrent sample cannot be lost to a smaller one - stale-idle: snapshot server stats after the refill wait so late creations are included in createdDelta/aliveAfter

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 47b6dd4224

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-08-17T06:59:35Z

+        } finally {
+            pool.shutdown(graceful = false)


Clear idle sandboxes before starting the next scenario

With the default --scenarios all, shutdown(graceful = false) stops pool workers but neither removes nor kills the pool's idle sandboxes, while the next scenario's mock.reset() clears only metrics and retains the sandbox registry. Consequently every later alive/aliveStats value includes all earlier pools—for example, resize.serverAliveAtEnd cannot describe the resized pool—and the advertised pool-size and over-creation trajectories are corrupted. Release each pool's idles or reset the mock registry between scenarios.

AGENTS.md reference: AGENTS.md:L43-L44

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-17T06:59:36Z

+                reconcileIntervalMs = 500,
+                idleTimeoutS = idleTimeoutS,


Disable proactive TTL reaping in the server-expiry scenario

Although the create handler now honors the 20-second timeout, fresh evidence shows this scenario still cannot exercise server-side expiry: because no explicit acquireMinRemainingTtl override is supplied, the SDK derives a 10-second threshold and PoolReconciler removes and kills these idles roughly halfway through their TTL. Thus serverKilled and serverCreatedDelta measure proactive client-side reaping rather than sandboxes expiring on the mock server; configure a zero threshold specifically for this scenario before claiming server-expiry behavior.

AGENTS.md reference: AGENTS.md:L28-L29

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-17T06:59:36Z

+        threads.shutdown()
+        // The loaders run for the full configured duration; the wait must not
+        // truncate them (a 15-min cap would silently halve a 30-min run).
+        threads.awaitTermination(durationMs / 1000 + 300, TimeUnit.SECONDS)


Cancel paced workers when the termination wait expires

Although the deadline recheck prevents late acquires, low pacing rates still leave non-daemon executor threads sleeping after this bounded awaitTermination returns, because its result is ignored and shutdownNow() is never called. For example, 16 workers at one acquire per minute reserve slots up to about 15 minutes out, but a 60-second scenario waits only 360 seconds before continuing; the report may finish while the JVM remains alive for many additional minutes. Cancel the remaining workers after the deadline or make pacing deadline-aware.

Useful? React with 👍 / 👎.

Pangjiping requested review from hittyt, jwx0925 and ninan-nn as code owners August 14, 2026 07:24

github-actions Bot added sdk/java sdks size/L Denotes a PR that changes 100-499 lines, ignoring generated files. documentation Improvements or additions to documentation labels Aug 14, 2026

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Pangjiping added 15 commits August 17, 2026 12:41

feat(benchmark): support production pool profile knobs (acquireMinRem…

95b5c60

…ainingTtl, primaryLockTtl, degradedThreshold) Map a production large-pool / high-frequency profile 1:1 onto driver options; document the example profile and scale caveats in README.

chore(benchmark): execd ping latency to fixed 100ms in the default pr…

e45ffc0

…ofile Keeps the high-frequency acquire case realistic without the 1-5s readiness probe capping the sustainable acquire rate.

docs(benchmark): fix stale default-profile comment in quick start

e2edb25

fix(benchmark): steady-state await must cover the full configured dur…

3b849c6

…ation awaitTermination(15min) silently truncated 30-min runs to ~15min. Wait for duration+5min and report the actual loader duration so achieved rate is computed against real runtime.

feat(benchmark): steady-start-immediately flag for startup-under-load…

e28aaaf

… scenario Loaders race the pool fill instead of waiting for idle; fixes the CLI parser for valueless boolean flags (previously swallowed the next --key as its value).

Pangjiping force-pushed the feat/pool-benchmark-harness branch from f367f7b to e28aaaf Compare August 17, 2026 05:01

github-actions Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 17, 2026

chatgpt-codex-connector Bot reviewed Aug 17, 2026

View reviewed changes

Uh oh!

Conversation

Pangjiping commented Aug 14, 2026

Summary

SDK companion

Evidence produced

Verification

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 17, 2026