Skip to content

fix: avoid sleeping while holding lifecycle locks - #19810

Merged
FrankChen021 merged 2 commits into
apache:masterfrom
FrankChen021:agent/codeql-sleep-with-lock
Sep 1, 2026
Merged

FrankChen021 merged 2 commits into
apache:masterfrom
FrankChen021:agent/codeql-sleep-with-lock

Conversation

@FrankChen021

Copy link
Copy Markdown
Member

What changed

  • move executor file-lock retry sleeps outside the lifecycle monitor while keeping channel creation, lock attempts, and lock publication synchronized
  • move GCE compute-service retry sleeps outside the autoscaler monitor
  • make the cached GCE service visible across threads and retain single initialization with a synchronized double check

Why

Both retry loops called Thread.sleep while holding an object monitor. A slow file-lock acquisition or delayed GCE service creation could therefore block unrelated lifecycle or autoscaler operations and triggered CodeQL's java/sleep-with-lock-held rule.

Impact

Retry timing and timeout behavior remain unchanged, but other threads no longer wait behind sleeping retry loops. Concurrent executor starts are still rejected, and the GCE compute service is still initialized at most once.

Validation

  • ExecutorLifecycleConfigTest (1 test)
  • GceAutoScalerTest (12 tests)
  • Maven validate / Checkstyle for indexing-service and extensions-contrib/gce-extensions
  • git diff --check

Fixes CodeQL alerts 2886 and 2887.

@FrankChen021 FrankChen021 changed the title Avoid sleeping while holding lifecycle locks fix: avoid sleeping while holding lifecycle locks Jul 30, 2026
@FrankChen021
FrankChen021 marked this pull request as ready for review July 30, 2026 22:27
Copilot AI review requested due to automatic review settings July 30, 2026 22:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses CodeQL’s java/sleep-with-lock-held findings by moving retry-loop sleeps out of synchronized/monitor-held regions in the executor lifecycle and GCE autoscaler paths, while preserving existing retry/timeout behavior and synchronization for critical state publication.

Changes:

  • Refactors ExecutorLifecycle file-lock acquisition into a helper that only synchronizes around tryLock()/publication and sleeps outside the monitor.
  • Updates GceAutoScaler compute-service initialization to avoid sleeping while holding the autoscaler monitor, using volatile + synchronized double-check for single initialization.
  • Adds unit tests covering lock retry success and immediate-timeout behavior for ExecutorLifecycle.acquireTaskFileLock(...).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
indexing-service/src/test/java/org/apache/druid/indexing/worker/executor/ExecutorLifecycleTest.java Adds coverage for file-lock retry and timeout paths.
indexing-service/src/main/java/org/apache/druid/indexing/worker/executor/ExecutorLifecycle.java Moves lock retry sleep out of the lifecycle monitor via a helper method.
extensions-contrib/gce-extensions/src/main/java/org/apache/druid/indexing/overlord/autoscaling/gce/GceAutoScaler.java Avoids sleeping while holding the autoscaler monitor; makes cached compute service cross-thread visible.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

extensions-contrib/gce-extensions/src/main/java/org/apache/druid/indexing/overlord/autoscaling/gce/GceAutoScaler.java:162

  • If createComputeServiceImpl() returns null on all attempts, createComputeService() will return null after the retry loop. All callers immediately dereference the result (e.g., computeService.instanceGroupManagers()), so this leads to a late NullPointerException rather than a clear initialization failure.

Consider throwing a GceServiceException after the retry loop if cachedComputeService is still null, to make the failure explicit and easier to diagnose.

        synchronized (this) {
          if (cachedComputeService == null) {
            cachedComputeService = createComputeServiceImpl();
          }
        }

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 3 of 3 changed files. No actionable correctness, concurrency, security, compatibility, or test-gap findings.


This is an automated review by Codex GPT-5.6-Sol

@FrankChen021
FrankChen021 merged commit 760f3cd into apache:master Sep 1, 2026
44 of 45 checks passed
@FrankChen021
FrankChen021 deleted the agent/codeql-sleep-with-lock branch September 1, 2026 12:40
@github-actions github-actions Bot added this to the 39.0.0 milestone Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants