Fix poller race condition that missed long-running builds - #18
Merged
Conversation
The BuildPoller used a watermark (highest ingested build ID) to skip already-seen builds. The AzDO API returned builds of any status, but only completed ones were ingested. When a long-running build was still in-progress, the watermark would advance past it via other completed builds with higher IDs. Once the long-running build finally completed, it was permanently skipped. Fix: - Add statusFilter=completed to GetBuildsForRepositoryAsync and GetRecentBuildsAsync so only terminal builds are returned - Replace watermark-based dedup with a simple DB existence check (FilterNewBuilds) -- builds already in the DB are skipped, new builds are always picked up regardless of ID ordering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Build 1475700 (roslyn-CI, a 75-minute PR build) completed but never appeared in Tiger. The root cause is a race condition in
BuildPoller: the AzDO API returned builds of any status, but only completed ones were ingested. The watermark (highest ingested build ID) would advance past long-running builds via other completed builds with higher IDs, permanently skipping them once they finished.Approach
Two changes work together to fix this:
statusFilter=completedon AzDO API calls --GetBuildsForRepositoryAsyncandGetRecentBuildsAsyncnow accept an optionalstatusFilterparameter. The poller passes"completed"so only terminal builds (succeeded, failed, canceled) are returned, never in-progress ones.Replace watermark with DB existence check -- Instead of tracking a high-water-mark build ID,
FilterNewBuildssimply checks whether each build already exists in the database. Builds already ingested are skipped; new builds are always picked up regardless of ID ordering. This eliminates the class of bugs where ID ordering doesn't match completion ordering.The
InsertBuildpath is unchanged --INSERT OR REPLACEfor the build row andINSERT OR IGNOREfor ingestion tasks remain idempotent.What's not included
Re-run detection (same build ID, new results after "Retry failed jobs") is intentionally deferred. It needs a dedicated design around storing build version info on ingestion tasks so stale results can be filtered at processing time.
Tests
FilterNewBuilds_NewBuildsAreIncluded-- new builds pass throughFilterNewBuilds_AlreadyIngestedBuildsAreSkipped-- duplicate builds are filteredFilterNewBuilds_LongRunningBuildNotMissed-- the exact scenario from the bug: build 150 completes after builds 100 and 200 are already ingested, and is correctly picked up