Skip to content

Fix concurrent S3 credential refresh and preloaded split stalls - #32

Draft
winningsix wants to merge 2 commits into
HighPerfDataAccelerator:devfrom
winningsix:fix/s3-credential-ready-splits-20260730
Draft

Fix concurrent S3 credential refresh and preloaded split stalls#32
winningsix wants to merge 2 commits into
HighPerfDataAccelerator:devfrom
winningsix:fix/s3-credential-ready-splits-20260730

Conversation

@winningsix

Copy link
Copy Markdown
Collaborator

Summary

  • consume asynchronously preloaded table-scan splits in ready order instead of
    binding a driver to the queue-front preload
  • wake all current split waiters after a preload completes so ready work cannot
    be stranded by a lost edge notification
  • wrap the default AWS credentials chain in an eager synchronized cache so
    concurrent S3 request signing shares one refresh while preserving normal
    credential rotation

Why

Two independent stalls appeared in cold S3 scans:

  1. Concurrent first use of WebIdentity credentials issued many redundant STS
    refreshes and produced an approximately 10-second tail.
  2. S3 range reads and H2D preparation completed, but 30 of 32 scan drivers
    remained blocked in kWaitForSplit. A preload completion only woke one
    waiter; notifications that arrived before waiters registered left ready
    splits queued with no later wakeup.

The second issue was reproduced with 495 splits per executor: before the fix,
only 357 splits were consumed before a 20-second query timeout. With wake-all,
the same cold query completed consistently.

Validation

  • velox_exec_infra_test --gtest_filter=TaskTest.preloadedSplitsAreConsumedInReadyOrder
  • velox_s3file_test --gtest_filter=S3UtilTest.synchronizedCachingCredentialsProvider
  • production build with warnings-as-errors
  • exact EMR 8.0 runtime native-load and relocation verification
  • SF1000 Q6, cold S3, 4 executors:
    • before: timeout at 20 seconds
    • after: 6.559 seconds and 7.175 seconds, both PASS

The S3 request-layer A/B used identical scan scheduling:

  • S3CrtClient, 8 MiB: 6.559–7.175 seconds
  • curl-multi, 8 MiB: 7.714 seconds
  • classic S3Client, 8 MiB: timeout at 20 seconds

Risk

  • preload completion wakes all current scan waiters. This is bounded by the
    task's scan-driver count and is required to preserve level-triggered
    readiness.
  • default credential acquisition moves to provider construction and refreshes
    five minutes before expiration. The underlying provider still owns
    credential discovery and rotation.

Avoid binding scan drivers to an arbitrary in-flight preload. Wake all current split waiters after each preload completion so accumulated ready work cannot be stranded by a lost edge notification.
Wrap the default AWS credential chain in an eager synchronized cache. Concurrent S3 signers share one refresh and continue to rotate credentials five minutes before expiration.
@github-actions

Copy link
Copy Markdown

Selective Build Plan

Linux release with adapters is running a full build (no cached dependency graph from main is available). See the CI workflows README for what this means.


Selective build plan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant