Source blob backups fail 100% with 'Source blob changed while it was being backed up' (ETag format mismatch)

[paullizer]

Summary

Every source blob in every Data Management backup fails with Source blob changed while it was being backed up. The failure rate is exactly 100% across all source blob containers, with zero retries, so no user document, group document, public document, or chat attachment has ever been backed up.

The job still reports completed_with_warnings, which makes the condition easy to miss.

Impact

• Severity: P0 — silent, total loss of source blob coverage in backups.
• Blast radius: every deployment with Enhanced Citations source blobs enabled.
• Also wastes substantial time and egress: the check runs after the blob is downloaded, encrypted, and uploaded, so the full transfer cost is paid and then discarded.
• Leaves orphaned pending artifacts in the backup container on every run.

Observed behavior

From job data_management_partial_20260818T0300Z:

Container	Blobs read	Copied	Failed
user-documents	71	0	71
group-documents	427	0	427
public-documents	95	0	95
personal-chat	19,394	0	19,394

[DATA_MANAGEMENT] Source blob backup completed with failures. -- {'job_id': 'data_management_partial_20260818T0300Z',
 'resource': 'source_blobs:personal-chat', 'container': 'personal-chat', 'failed_count': 19394,
 'copied_count': 0, 'skipped_count': 0, 'source_read_count': 19394,
 'failure_reasons': '19394x Source blob changed while it was being backed up.'}

Job telemetry reports Retries / throttles: 0 / 0, confirming a non-transient first-attempt failure.

Root cause

_transfer_backup_source_blob compares two ETags that come from different Azure SDK code paths with different transport formats:

current_source_properties = source_blob_client.get_blob_properties()
current_source_etag = _safe_text(_get_backup_blob_property(current_source_properties, "etag"))
if source_etag and current_source_etag != source_etag:
    raise RuntimeError("Source blob changed while it was being backed up.")

Value	Origin	SDK path	Format
source_etag	list_blobs()	get_blob_properties_from_generated_code() reads the XML <Etag> element	0x8DE...
current_source_etag	get_blob_properties()	BlobProperties(**headers) reads the HTTP ETag header	"0x8DE..."

The HTTP ETag header is an RFC 7232 quoted-string; the List Blobs XML element is not. azure-storage-blob==12.24.1 performs no normalization in either direction, so the comparison is always unequal.

RuntimeError is not retryable under _is_retryable_backup_blob_error, so every blob fails on the first attempt.

Independent confirmation that the blobs did not actually change: the ranged reads send that same unquoted ETag as an If-Match precondition and Azure accepts them. Only the Python-side string comparison is wrong.

Secondary problem: one Cosmos write per blob

record_transfer_result calls persist() unconditionally for every item, producing one Cosmos checkpoint write per blob — 19,394 writes for personal-chat alone. This caps throughput at roughly 6 items/second and accounts for the 74-minute runtime. It will remain a bottleneck once blobs start succeeding.

Expected behavior

• Source blobs are backed up successfully.
• A genuine mid-transfer source change is still detected and rejected.
• Checkpoint writes are batched so throughput is not bound by a Cosmos round-trip per item.

Proposed fix

• Normalize both operands before comparing: strip transport quoting and the optional W/ weak-validator prefix. Keep source_item["source_etag"] exactly as returned by list_blobs() so the If-Match precondition sent on ranged reads is unchanged.
• Batch checkpoints on manifest batch size (100) or a 15 second interval, whichever comes first, while still asserting the job lease on every item.

The migration path has a visually similar comparison but sources both operands from get_blob_properties(), so it is not affected.

Validation

New functional test functional_tests/test_data_management_backup_source_blob_etag.py exercises the real transfer path with an unquoted listing and a quoted fetch. Neutralizing the normalization reproduces the exact production message:

Transfer must succeed, got 'failed' ('Source blob changed while it was being backed up.')

Environment

• SimpleChat 0.250.217
• azure-storage-blob==12.24.1
• Azure App Service, 3 instances

[paullizer]

Shipped in v0.250.220 via #1273.

Release note entries:

• v0.250.220: Source Blob Backup ETag Failure
• v0.250.220: Source Blob Backup Checkpoint Throughput
• v0.250.220: Functional Tests Silently Passing Under pytest

Full notes: https://microsoft.github.io/simplechat/explanation/release_notes/

Closing as completed. This issue stayed open after the fix merged because pull requests target Development while the repository default branch is main, so GitHub never applied the Fixes # keyword automatically.