Admin settings can revert across workers after successful saves

[Bionic711]

Issue

Admin settings can appear to revert after a successful save in a scaled-out, containerized App Service deployment. Reloads can alternate between the old and saved settings without another admin save. Some code paths can also overwrite the persisted settings with an older snapshot.

Steps to Reproduce

1. Use multiple Python workers or app instances sharing settings storage.
2. Load Admin Settings so more than one worker has read the existing settings.
3. Change and save a setting.
4. Reload through different workers. Observe old -> saved -> old values before local version TTLs expire.
5. For the lost-update case, have a stale worker update unrelated metadata (for example, the release-check timestamp). Its full-document upsert can revert the setting in Cosmos.

Expected Behavior

A completed save should be visible to other workers without waiting for local settings cache expiry. Unrelated settings writes, startup initialization, and recovery from Redis failures must not restore older values. Failed or unconfirmed saves must not be reported as successful.

Confirmed Code Causes

• Worker-local settings snapshots reuse a cached shared version for 15 seconds, even with Redis enabled.
• update_settings reads through that cache and upserts the entire document without an ETag precondition.
• Settings reads/migrations and auxiliary metadata writers can persist stale documents.
• Worker/scheduler startup publishes an earlier snapshot to Redis without coordinated versioning.
• Redis outage fallback advances a separate Cosmos version while Redis retains its old payload/version.
• The 302 redirect occurs after synchronous save/cache calls return; adding a delay is not the root-cause fix.

These paths were reproduced offline with independent worker modules and shared fake services. They establish code defects, but do not identify which path occurred on the reported live deployment.

Agreed Behavior and Local Implementation

Implemented locally on branch fix/admin-settings-consistency, version 0.261.025; not committed, pushed, or deployed yet.

• Remove worker-local app settings payload/version caches. Read shared Redis on each lookup, or Cosmos directly when Redis is disabled.
• Reject settings saves when configured Redis is unavailable; allow Cosmos read fallback. Do not return an old worker snapshot when both services fail.
• Use ETag-conditional Cosmos writes, partial updates, and a document revision.
• Publish settings, revision, and the Cosmos session token together using a single Redis state key and atomic compare-and-set.
• Persist a pending marker before a database write so interrupted publication cannot expose the previous Redis payload. Repair abandoned publication after its 30-second lease via a conditional Cosmos write.
• Remove startup snapshot publication and add an admin form revision check.
• Make auxiliary writers use intended field updates and report failed saves accurately.
• Complete the logging-settings re-entrancy guard so cache failure logging cannot recurse.

Acceptance Criteria

• Remove the 15-second worker-local settings staleness window without changing unrelated caches.
• Prevent stale snapshots and concurrent metadata updates from overwriting unrelated saved settings.
• Prevent delayed startup or expired writers from publishing older cache contents.
• Reject writes before database mutation when configured Redis is unavailable.
• Surface an unconfirmed result if Redis fails after a database write; never imply a guaranteed rollback.
• Cover Redis failure/recovery, no-Redis reads, migrations, creation, partial writes, stale forms, and logging re-entrancy with deterministic tests.
• Commit/review the implementation and deploy all participating workers and scheduler together.
• Validate repeated saves/reloads across the scaled-out deployment and confirm instance/process/revision behavior.

Validation

• Combined targeted tests: 190 passed, 1 skipped. The skipped test requires an authenticated browser environment.
• Syntax checked for 27 changed/new Python files; existing added-line XSS/access-control checks reported zero findings; whitespace checks clean.
• Standalone logging/merge regressions and documentation site quality passed.
• Documentation coverage has an unrelated pre-existing inventory mismatch in the chat upload label (xsd); left unchanged.

Notes

• Cosmos fallback remains subject to Session consistency; removing local cache does not create global strong consistency during outages.
• Coordinated rollout is required: old workers do not participate in the new publication protocol. Redis connection/enablement changes require coordinated worker restarts.
• Relevant files: app_settings_store.py, app_settings_cache.py, functions_settings.py, functions_appinsights.py, route_frontend_admin_settings.py, worker/scheduler startup, auxiliary settings writers.
• Tests: functional_tests/test_app_settings_store_consistency.py, functional_tests/test_app_settings_auxiliary_writers.py, ui_tests/test_admin_settings_save_consistency.py.
• Documentation: docs/admin/scale.md; release notes for v0.261.025.
• Priority: P1. Estimated size: L (shared storage protocol, multiple writer surfaces, and concurrency/failure validation).

[Bionic711]

Follow-up: Redis Explorer regression corrected locally (v0.261.026)

After deploying the settings consistency changes, Redis Explorer key requests returned HTTP 500. Existing Explorer tests reproduced an AttributeError: _resolve_redis_keys() still referenced app_settings_cache.APP_SETTINGS_CACHE_KEY and APP_SETTINGS_CACHE_VERSION_KEY, which the cache removal had deleted.

Correction

• Classify the new shared settings record using app_settings_store.SETTINGS_STATE_KEY.
• Recognize the old payload/version names as legacy keys within Explorer, without restoring worker-local settings caching.
• Preserve sensitive nested-field and Cosmos session-token redaction for ready/pending shared settings records.
• Updated app version and release notes to 0.261.026.

Validation

• 62 targeted tests passed, covering Redis Explorer, service detection, client factory, authentication, and settings consistency.
• Explorer's standalone runner passed 10/10 tests, including eight service/auth/client combinations:
  • Azure Cache for Redis: TLS 6380.
  • Azure Managed Redis: TLS 10000.
  • Key and managed-identity authentication for each.
  • App-cache and session-client fallback for each.
• Matrix covers key pagination, new/legacy key classification, safe value previews, and omitted optional INFO metrics.
• Existing XSS/access-control guardrails and whitespace checks passed.

Tests use actual application/factory code with simulated Redis I/O, not live end-to-end tests against both Azure services. The correction is local on fix/admin-settings-consistency and has not been committed, pushed, or redeployed. Live Explorer validation is still required after redeployment.

[Bionic711]

PR #1478 CodeQL remediation (v0.261.027)

Pushed commit 3faf396 to the existing draft PR.

• Removed reverse cache imports into config/settings/logging/Redis-factory modules. Configure from the passed settings object, with runtime container handles/factory/logging callbacks supplied separately by the settings and startup owners.
• Updated both web and scheduler bootstrap wiring. Redis-required writes still fail closed before client configuration or on construction failure; no worker-local settings snapshot restored.
• Removed state-changing operations from test assertions, including the flagged migration write.
• Strengthened Python, functional-test, and PR-preparation instructions: trace function-local import edges, do not treat lazy imports as a cycle fix, exercise real-module startup, restore test stubs, and inspect CodeQL annotations in addition to workflow conclusions.
• Added normal/-O fresh-process import/bootstrap probes with configuration imports and network access blocked. Version and release notes updated to 0.261.027.

Validation: 209 focused tests passed, 1 authenticated UI test skipped; standalone extraction (19 cases), logging, and merge regressions passed. All workflow-covered application modules plus changed tests compiled (230 files). Full-file XSS/BAC, Swagger and whitespace checks passed. Dependency graph verification found no cache path to config/settings/logging/Redis factory and no newly introduced cyclic edge.

Hosted verification completed: CodeQL run 34398785015 succeeded on final commit 4b0c836, including the one-line unused-import cleanup after 3faf396. Alerts #2732, #2733, and #2734 are fixed; the PR merge ref has zero open CodeQL findings, all review threads are resolved, and all PR checks passed. PR #1478 remains a draft. No Azure deployment or configuration change performed.