Skip to content

Add an offline media byte budget and authenticated cache monitoring - #20

Merged
sidinsearch merged 1 commit into
sidinsearch:experimental/video-playbackfrom
RahulBabu06:codex/media-cache-budget
Sep 21, 2026
Merged

sidinsearch merged 1 commit into
sidinsearch:experimental/video-playbackfrom
RahulBabu06:codex/media-cache-budget

Conversation

@RahulBabu06

Copy link
Copy Markdown

Age-based retention can leave large volumes of recently downloaded MP4s on disk.
This adds MEDIA_MAX_BYTES: after expiry and orphan cleanup, each sweep evicts
the oldest finalized MP4s until usage is within the configured budget. The default
is 0 (disabled); invalid values warn and fall back to zero. For example,
MEDIA_MAX_BYTES=10737418240 targets a 10 GiB MP4 cache.

Eviction uses the existing cross-process maintenance lock, updates media metadata
for delta sync, and preserves saved analyses and device copies. Newly published,
unreferenced files retain their one-hour grace period. Failed deletions are
attempted once per sweep while other candidates remain eligible; any remaining
excess is reported as over_budget_bytes and logged.

Operators can inspect the cache with GET /admin/media-cache/stats and trigger
cleanup with POST /admin/media-cache/sweep. Both require the existing access
token and run filesystem/database work off the event loop. Statistics include
file count, bytes, oldest/newest ages, budget excess, and the last completed sweep.
Sweep history persists across API workers and restarts. Unavailable operations
return 503; completed sweeps report individual deletion failures in their result.

Relative MEDIA_PATH values now resolve from the backend directory so the API
and download subprocesses use the same location regardless of their working
directory. Existing deployments using a relative path from another launch
directory should set an absolute path to preserve their cache location.

This is a budget applied at each sweep. It excludes active download workspaces
and other application storage, and usage can exceed it between sweeps or when
files cannot be evicted. At 50 MB/video, 1,000 videos use about 50 GB, and
100 videos/day over 14 days use about 70 GB before cleanup. Operators should allow
temporary-download headroom and use a volume quota when an absolute cap is needed.
Configuration, monitoring requests, and capacity guidance are documented in
docs/OFFLINE_MEDIA.md, the environment example, and Docker Compose.

Validation:

  • Focused backend suite: 65 tests pass locally on Python 3.11, covering metadata,
    retention, media/admin endpoints, temporary
    download cleanup, and delta-sync pagination. Includes budget boundaries,
    eviction order, failed deletions, publication grace, persisted statistics,
    cross-process coordination, authentication, and relative-path consistency.
  • Client: 33 Jest tests across five suites pass; application and test TypeScript
    checks pass.
  • GitHub CI remains required before promotion to beta. Native playback and live
    provider downloads remain release smoke tests.

@sidinsearch
sidinsearch changed the base branch from main to experimental/video-playback September 21, 2026 16:53
@sidinsearch

sidinsearch commented Sep 21, 2026 •

Copy link
Copy Markdown
Owner

Heads-up: base changed from main → experimental/video-playback

Nice work — this directly addresses the two follow-up gaps I flagged on #19:

  • MEDIA_MAX_BYTES for total-size capping (default 0 = disabled)
  • GET /admin/media-cache/stats + POST /admin/media-cache/sweep (auth-protected)

All 5 checks pass on the original target (main), but I changed the base to experimental/video-playback before merging to keep the staged rollout path we agreed on.

PR #13 ─┐
PR #19 ─┼─► experimental/video-playback  (held here — no current plan to merge into beta/main)
PR #20 ─┘

Status: kept as experimental. There's no plan to promote experimental/video-playback → beta → main right now. The offline media pipeline is held here for evaluation and iteration only.

Next time: target experimental/video-playback directly (or a feature branch off it). Targeting main skips the staged-rollout guard and risks landing experimental work in production.


What this PR adds

  1. MEDIA_MAX_BYTES env var — media_retention.py adds non_negative_env_int helper. After age/orphan cleanup, sweep evicts oldest-first until total_bytes <= max_bytes. Respects grace period for unreferenced fresh files. Skips files that already failed deletion in this sweep.
  2. get_media_cache_stats(media_dir) — returns {file_count, total_bytes, oldest_age_days, newest_age_days, last_sweep_at, max_bytes, over_budget_bytes, last_sweep_result}. Uses the same .maintenance.sqlite3 lock so it sees consistent state across API workers. Reads last_sweep_at + result_json from a new media_cache_sweep_state table — shared by all workers and survives restarts.
  3. GET /admin/media-cache/stats — token-protected, runs in asyncio.to_thread, returns 503 on failure (not 500).
  4. POST /admin/media-cache/sweep — runs the configured retention + size policy on demand. Same auth + 503 handling.
  5. Persistence of sweep history — _record_sweep() writes to media_cache_sweep_state (singleton row, UPSERT) so the operator endpoint shows what the last sweep did.
  6. Relative MEDIA_PATH consistency — now resolves from backend dir, not CWD. (Operators using a relative path from a different launch dir should set an absolute path.)

Test coverage

  • 65 backend tests pass locally (Python 3.11) — including budget boundaries, eviction order, failed deletions, publication grace, persisted statistics, cross-process coordination, auth, relative-path consistency.
  • 33 client Jest tests across 5 suites pass.
  • GitHub CI: backend (unittest), client (npm test + typecheck), Backend Tests (pytest), Build and Push Docker Image — all green.

Minor observations (not blockers)

  1. Quota eviction loop continues even if total_bytes > max_bytes after a failed unlink — the continue correctly moves to next file but doesn't bump over_budget_bytes for visibility. Could be added in a follow-up.
  2. Admin endpoints are token-protected, not role-protected — anyone with the access token can hit them. For single-user deployments this is fine (same threat model as /api/v1/media/*). For multi-tenant deployments consider a separate ADMIN_TOKEN. Worth documenting in docs/DEPLOYMENT.md.
  3. _record_sweep writes inside the same lock as the eviction loop — good for consistency, but means a slow DB write blocks the next sweep start. Fine at default 1-hour interval.
  4. MEDIA_MAX_BYTES=0 semantics are documented as "disabled" — but non_negative_env_int returns default=0 for invalid values too. So an operator who fat-fingers the value silently disables the budget. Could log an ERROR instead of WARNING for invalid input.

Summary

Held as experimental — no plan to merge into beta or main at this time. This branch exists for evaluation, iteration, and discussion. Future promotion (if any) will be a deliberate decision, not an automatic merge.

@sidinsearch
sidinsearch merged commit 11e77a8 into sidinsearch:experimental/video-playback Sep 21, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants