Summary
GET /api/admin/data-management/backups returns 503 and the Backup Inventory panel shows the generic "Data Management history could not be loaded. Please try again later or review application logs."
The underlying provider error is caught but its status code and message are discarded, so the logs cannot identify the cause. Six rounds of investigation across App Service console logs, App Insights, Cosmos Maintenance, and Cosmos diagnostic logs failed to determine why — because the one component that had the full error in hand threw it away.
Impact
- Backup Inventory and Job History panels are unusable when this fires.
- No data loss, but the operator has no actionable path — the message literally says "review application logs" and the logs contain nothing useful.
- Every failure mode collapses to one indistinguishable 503.
Observed behavior
Browser console:
/api/admin/data-management/backups?status=available&scheduled=all&page_size=25
Failed to load resource: the server responded with a status of 503 (Service Unavailable)
App Service console log — this is the entire diagnostic output:
[DEBUG][ERROR][Log] [DATA_MANAGEMENT] Data Management history could not be loaded. --
{'history_list': 'backups', 'reason': 'history_provider_unavailable',
'maintenance_required': False, 'error_type': 'CosmosHttpResponseError'}
error_type is the exception class name only. No status code, no provider message.
What has been ruled out
| Candidate |
Status |
Evidence |
| Missing composite index |
Ruled out |
Admin Settings → Scale → Cosmos Maintenance reports Indexing Policy Status Aligned, 7 containers checked, Missing Expected Indexes: 0. data_management_jobs is one of the checked containers. |
Parameterized TOP unsupported |
Ruled out |
SELECT TOP @parameter is valid Cosmos NoSQL. |
| Cosmos diagnostic 400s |
Red herring |
400 + requestCharge 0 + sub-ms duration is the normal cross-partition query-plan negotiation. Present on every container (tabular_export_runs, settings, data_management_jobs) while those code paths log success. |
Remaining candidates are throttling (429) or another non-400 CosmosHttpResponseError. data_management_jobs was observed oscillating 1,000→5,000 RU on container_utilization_above_threshold, and Admin Settings fires a burst of Cosmos-heavy admin calls on load.
Root cause of the diagnosis failure
_data_management_history_unavailable_response in route_backend_data_management.py logs only type(original_error).__name__. _raise_data_management_history_unavailable in functions_data_management.py discards the provider status and message entirely when constructing DataManagementHistoryUnavailableError.
Additionally:
_is_data_management_history_index_error requires status 400 and the literal text "composite index". Any wording drift routes to the generic message.- Throttling has no dedicated classification, so a
429 produces "review application logs" instead of "retry shortly".
_query_data_management_history_items has no retry of its own beyond the SDK default.
Expected behavior
- The provider status code and sanitized message reach operator logs.
- Throttled reads retry briefly and, if still failing, report retryable busy guidance rather than a generic error.
- Index errors are recognized across reasonable provider wording variations.
Proposed fix
- Carry
provider_status_code and provider_message on DataManagementHistoryUnavailableError and log both. Provider detail stays out of the browser payload.
- Add throttle classification (
429/503, "request rate is large") with a distinct retryable message.
- Bounded retry on the history query for throttles and transient transport errors.
- Loosen the index classifier to also match
ORDER BY + "does not have a corresponding" / "not served".
This does not require knowing the root cause and makes the next occurrence self-diagnosing.
Environment
- SimpleChat
0.250.219
- Azure Cosmos DB, container-level autoscale
- Azure App Service, 3 instances
Summary
GET /api/admin/data-management/backupsreturns 503 and the Backup Inventory panel shows the generic "Data Management history could not be loaded. Please try again later or review application logs."The underlying provider error is caught but its status code and message are discarded, so the logs cannot identify the cause. Six rounds of investigation across App Service console logs, App Insights, Cosmos Maintenance, and Cosmos diagnostic logs failed to determine why — because the one component that had the full error in hand threw it away.
Impact
Observed behavior
Browser console:
App Service console log — this is the entire diagnostic output:
error_typeis the exception class name only. No status code, no provider message.What has been ruled out
data_management_jobsis one of the checked containers.TOPunsupportedSELECT TOP @parameteris valid Cosmos NoSQL.400+requestCharge 0+ sub-ms duration is the normal cross-partition query-plan negotiation. Present on every container (tabular_export_runs,settings,data_management_jobs) while those code paths log success.Remaining candidates are throttling (
429) or another non-400CosmosHttpResponseError.data_management_jobswas observed oscillating 1,000→5,000 RU oncontainer_utilization_above_threshold, and Admin Settings fires a burst of Cosmos-heavy admin calls on load.Root cause of the diagnosis failure
_data_management_history_unavailable_responseinroute_backend_data_management.pylogs onlytype(original_error).__name__._raise_data_management_history_unavailableinfunctions_data_management.pydiscards the provider status and message entirely when constructingDataManagementHistoryUnavailableError.Additionally:
_is_data_management_history_index_errorrequires status400and the literal text"composite index". Any wording drift routes to the generic message.429produces "review application logs" instead of "retry shortly"._query_data_management_history_itemshas no retry of its own beyond the SDK default.Expected behavior
Proposed fix
provider_status_codeandprovider_messageonDataManagementHistoryUnavailableErrorand log both. Provider detail stays out of the browser payload.429/503, "request rate is large") with a distinct retryable message.ORDER BY+ "does not have a corresponding" / "not served".This does not require knowing the root cause and makes the next occurrence self-diagnosing.
Environment
0.250.219