CI: search unit tests fail repo-wide once mmlspark-azure-search hits its 50-index cap, and cannot self-recover #2639

Summary

UnitTests misc, UnitTests search1 and UnitTests search2 fail on master and on every open PR once the shared mmlspark-azure-search service reaches its 50-index cap. The failure is not self-healing: once the cap is hit, it stays hit until someone deletes indexes by hand.

I hit this while working on unrelated PRs and cleared it manually (deleted 18 abandoned test-* indexes, 48/50 → 35/50), so CI is unblocked right now. It will recur.

Evidence

refs/heads/master build 231241262 — no PR changes — failed on UnitTests search1, UnitTests search2, UnitTests misc with:

HTTP/1.1 429 Too Many Requests
requestUrl: https://mmlspark-azure-search.search.windows.net/indexes?api-version=...
responseBody: {"error":{"message":"An index with the name 'test--1326844747-20260815003139417'
  in service 'mmlspark-azure-search' could not be created. Index quota has been exceeded for
  this service. You must either delete unused indexes first, add a delay between index creation
  requests, or upgrade the service for higher limits. Maximum number of indexes allowed: 50"}}

Every build queued after ~22:54 UTC on 2026-08-14 failed this way; every build before it passed.

The suite reports Tests: succeeded 0, failed 0 — it aborts before a single test runs.

Root cause

All in cognitive/src/test/scala/.../search/split1/SearchWriterSuitePart1.scala, the shared base for both the search1 and search2 jobs.

1. The retention window is longer than the time it takes to exhaust the quota. (the primary defect)

val twoDaysAgo = LocalDateTime.now().minusDays(2)   // line 208

cleanOldIndexes() only deletes indexes older than 2 days. When I inspected the service it held 45 leaked test-* indexes and the oldest was 40.8 h old — every one of them was inside the 48 h retention window, so the sweep selected exactly zero. At current CI volume the service reaches 50 indexes in roughly 40 h, i.e. well before anything becomes eligible for collection. Retention keyed purely on age can never keep up with a cap that fills faster than the window expires.

2. Cleanup aborts at the first failure, stranding everything behind it.

val successfulCleanup = ... .map(n => deleteIndex(n)).forall(_ == 204)
assert(successfulCleanup)

assert throws out of cleanTestIndices, which is called from afterAll() before cleanOldIndexes() and super.afterAll() — so one bad delete (or a transient 5xx) skips the sweep entirely and leaks every index the run created. forall on a strict Seq does attempt all the deletes, but the assert still discards the sweep that follows it.

Correction to an earlier version of this issue

An earlier revision claimed cleanOldIndexes() was only reachable from afterAll() and therefore unreachable once beforeAll() started throwing. That is wrong — beforeAll() on master already calls cleanOldIndexes() first:

override def beforeAll(): Unit = {
  // to ensure that we clean up old indexes at the start of the test
  // so we have enough capacity to create new ones
  cleanOldIndexes()
  ...

The recovery path exists and runs. It simply selected nothing, for the reason in (1). Leaving the incorrect claim here would have sent a fix in the wrong direction.

Fix

Opened a PR: retention becomes pressure-driven rather than age-only.

• Anything past a 12 h routine age is always collected.
• When that still doesn't leave 10 free slots, keep taking the next-oldest index until it does.
• A 3 h floor is never crossed, so an index a concurrently running build could still own is never touched (a full pipeline run finishes well inside an hour).
• Deletes no longer abort the sweep; failures are collected and reported, 404 counts as success, and afterAll is wrapped so the sweep and super.afterAll() always run.

The policy is extracted into a pure object with unit tests that need no search credentials, so it is covered on every build — this bug took down master, so its regression test must not depend on the secrets that gate the search jobs.

Related, deliberately out of scope

cognitive/src/test/scala/.../face/FaceSuite.scala:282 has the same 2-day anti-pattern for a different resource. Not touched here.