Fix Data Management history failure diagnostics and throttle handling

[Paul Lizer (paullizer)]

Fixes #1275

Problem

GET /api/admin/data-management/backups returns 503 and Backup Inventory shows "Data Management history could not be loaded. Please try again later or review application logs."

Reviewing the application logs is not actionable. This is the entire diagnostic output:

[DEBUG][ERROR][Log] [DATA_MANAGEMENT] Data Management history could not be loaded. --
 {'history_list': 'backups', 'reason': 'history_provider_unavailable',
  'maintenance_required': False, 'error_type': 'CosmosHttpResponseError'}

error_type is the exception class name. No status code, no provider message. Six rounds of investigation across App Service console logs, App Insights, Cosmos Maintenance, and Cosmos diagnostic logs could not determine the cause — because the component that caught the error with full detail discarded it.

Root cause of the diagnosis failure

_raise_data_management_history_unavailable constructed DataManagementHistoryUnavailableError without retaining the provider status or message, and _data_management_history_unavailable_response logged only type(original_error).__name__.

Classification was also too narrow — _is_data_management_history_index_error required status 400 and the literal substring "composite index", so throttling and every other provider failure collapsed into the same opaque 503.

Ruled out during investigation

Candidate	Outcome	Evidence
Missing composite index	Ruled out	Cosmos Maintenance: Indexing Policy Status Aligned, 7 containers checked, Missing Expected Indexes: 0. data_management_jobs is one of the checked containers.
Parameterized TOP unsupported	Ruled out	SELECT TOP @parameter is valid Cosmos NoSQL.
The 400s in Cosmos diagnostics	Red herring	400 + requestCharge 0 + sub-ms duration is the normal cross-partition query-plan negotiation. It appears on tabular_export_runs and settings too, while those code paths log success.

Remaining candidates are throttling or another non-400 CosmosHttpResponseError. data_management_jobs was observed oscillating 1,000→5,000 RU on container_utilization_above_threshold, and Admin Settings fires a burst of Cosmos-heavy admin calls on load.

Changes

1. Retain provider detail — DataManagementHistoryUnavailableError now carries provider_status_code, provider_message, and retryable. The route logs status_code and error.

Provider text is confined to operator logs and never enters the browser payload. safe_message remains a fixed, non-reflective string, so there is no new information-exposure surface.

2. Throttle classification — 429/503 or "request rate is large" / "too many requests" now raises DATA_MANAGEMENT_HISTORY_BUSY_MESSAGE with retryable=True, and the response carries retryable: true so the UI can offer a retry instead of pointing at logs.

3. Bounded retry — _query_data_management_history_items retries up to 3 times for throttled and transient transport errors with jittered backoff capped at 4 seconds, logging attempt, status code, delay, and provider message. Non-retryable errors still fail on the first attempt.

4. Broader index detection — still requires 400, but now also matches ORDER BY combined with "does not have a corresponding" / "not served", so maintenance guidance survives provider wording drift.

Validation

12 passed

Test	Assertion
test_history_throttling_is_retried_then_reported_as_busy	429 retries exactly DATA_MANAGEMENT_HISTORY_QUERY_MAX_ATTEMPTS times, then reports history_provider_throttled with retryable=True; provider text absent from safe_message
test_history_failures_capture_provider_detail_for_operator_logs	provider_status_code / provider_message populated, absent from safe_message; a non-retryable 403 does not retry
test_history_index_errors_match_alternate_provider_wording	Index guidance triggers without the word "composite"
test_history_provider_index_errors_are_actionable	Existing coverage unchanged

Regression probe: neutralizing _is_data_management_history_throttle_error fails the throttle test, confirming it exercises the new classification rather than passing incidentally.

Full Data Management suite: 150 passed, 1 failed — test_backup_recovery_and_admin_progress_are_bounded_and_sanitized, pre-existing on origin/Development and unrelated.

Drive-by test fix

test_deployers_apply_the_data_management_history_index asserted deployer_version == "1.0.24" and started failing when deployers/version.txt advanced to 1.0.25. That is the exact brittle pattern the repository instructions prohibit for version assertions. Now uses compare_simplechat_versions(deployer_version, "1.0.24") >= 0, preserving intent without breaking on future bumps.

Honest scope note

This does not fix the underlying provider failure, which is still unconfirmed. It makes the next occurrence self-diagnosing — status code and message land in one log line. If it turns out to be throttling, the bounded retry here may resolve it outright; the checkpoint batching in 0.250.218 also reduces sustained write pressure on the same containers.

Version

0.250.219 → 0.250.220

Documentation

• docs/explanation/fixes/DATA_MANAGEMENT_HISTORY_DIAGNOSTICS_FIX.md
• docs/explanation/release_notes.md

[Paul Lizer (paullizer)]

Copilot resolve the merge conflicts in this pull request

[Copilot]

Copilot resolve the merge conflicts in this pull request

Done — I merged origin/Development into this branch and resolved the release-notes conflict in commit 04ced07f.