You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add an offline media byte budget and authenticated cache monitoring - #20
Age-based retention can leave large volumes of recently downloaded MP4s on disk.
This adds MEDIA_MAX_BYTES: after expiry and orphan cleanup, each sweep evicts
the oldest finalized MP4s until usage is within the configured budget. The default
is 0 (disabled); invalid values warn and fall back to zero. For example, MEDIA_MAX_BYTES=10737418240 targets a 10 GiB MP4 cache.
Eviction uses the existing cross-process maintenance lock, updates media metadata
for delta sync, and preserves saved analyses and device copies. Newly published,
unreferenced files retain their one-hour grace period. Failed deletions are
attempted once per sweep while other candidates remain eligible; any remaining
excess is reported as over_budget_bytes and logged.
Operators can inspect the cache with GET /admin/media-cache/stats and trigger
cleanup with POST /admin/media-cache/sweep. Both require the existing access
token and run filesystem/database work off the event loop. Statistics include
file count, bytes, oldest/newest ages, budget excess, and the last completed sweep.
Sweep history persists across API workers and restarts. Unavailable operations
return 503; completed sweeps report individual deletion failures in their result.
Relative MEDIA_PATH values now resolve from the backend directory so the API
and download subprocesses use the same location regardless of their working
directory. Existing deployments using a relative path from another launch
directory should set an absolute path to preserve their cache location.
This is a budget applied at each sweep. It excludes active download workspaces
and other application storage, and usage can exceed it between sweeps or when
files cannot be evicted. At 50 MB/video, 1,000 videos use about 50 GB, and
100 videos/day over 14 days use about 70 GB before cleanup. Operators should allow
temporary-download headroom and use a volume quota when an absolute cap is needed.
Configuration, monitoring requests, and capacity guidance are documented in docs/OFFLINE_MEDIA.md, the environment example, and Docker Compose.
Validation:
Focused backend suite: 65 tests pass locally on Python 3.11, covering metadata,
retention, media/admin endpoints, temporary
download cleanup, and delta-sync pagination. Includes budget boundaries,
eviction order, failed deletions, publication grace, persisted statistics,
cross-process coordination, authentication, and relative-path consistency.
Client: 33 Jest tests across five suites pass; application and test TypeScript
checks pass.
GitHub CI remains required before promotion to beta. Native playback and live
provider downloads remain release smoke tests.
Heads-up: base changed from main → experimental/video-playback
Nice work — this directly addresses the two follow-up gaps I flagged on #19:
MEDIA_MAX_BYTES for total-size capping (default 0 = disabled)
GET /admin/media-cache/stats + POST /admin/media-cache/sweep (auth-protected)
All 5 checks pass on the original target (main), but I changed the base to experimental/video-playback before merging to keep the staged rollout path we agreed on.
PR #13 ─┐
PR #19 ─┼─► experimental/video-playback (held here — no current plan to merge into beta/main)
PR #20 ─┘
Status: kept as experimental. There's no plan to promote experimental/video-playback → beta → main right now. The offline media pipeline is held here for evaluation and iteration only.
Next time: target experimental/video-playback directly (or a feature branch off it). Targeting main skips the staged-rollout guard and risks landing experimental work in production.
What this PR adds
MEDIA_MAX_BYTES env var — media_retention.py adds non_negative_env_int helper. After age/orphan cleanup, sweep evicts oldest-first until total_bytes <= max_bytes. Respects grace period for unreferenced fresh files. Skips files that already failed deletion in this sweep.
get_media_cache_stats(media_dir) — returns {file_count, total_bytes, oldest_age_days, newest_age_days, last_sweep_at, max_bytes, over_budget_bytes, last_sweep_result}. Uses the same .maintenance.sqlite3 lock so it sees consistent state across API workers. Reads last_sweep_at + result_json from a new media_cache_sweep_state table — shared by all workers and survives restarts.
GET /admin/media-cache/stats — token-protected, runs in asyncio.to_thread, returns 503 on failure (not 500).
POST /admin/media-cache/sweep — runs the configured retention + size policy on demand. Same auth + 503 handling.
Persistence of sweep history — _record_sweep() writes to media_cache_sweep_state (singleton row, UPSERT) so the operator endpoint shows what the last sweep did.
Relative MEDIA_PATH consistency — now resolves from backend dir, not CWD. (Operators using a relative path from a different launch dir should set an absolute path.)
GitHub CI: backend (unittest), client (npm test + typecheck), Backend Tests (pytest), Build and Push Docker Image — all green.
Minor observations (not blockers)
Quota eviction loop continues even if total_bytes > max_bytes after a failed unlink — the continue correctly moves to next file but doesn't bump over_budget_bytes for visibility. Could be added in a follow-up.
Admin endpoints are token-protected, not role-protected — anyone with the access token can hit them. For single-user deployments this is fine (same threat model as /api/v1/media/*). For multi-tenant deployments consider a separate ADMIN_TOKEN. Worth documenting in docs/DEPLOYMENT.md.
_record_sweep writes inside the same lock as the eviction loop — good for consistency, but means a slow DB write blocks the next sweep start. Fine at default 1-hour interval.
MEDIA_MAX_BYTES=0 semantics are documented as "disabled" — but non_negative_env_int returns default=0 for invalid values too. So an operator who fat-fingers the value silently disables the budget. Could log an ERROR instead of WARNING for invalid input.
Summary
Held as experimental — no plan to merge into beta or main at this time. This branch exists for evaluation, iteration, and discussion. Future promotion (if any) will be a deliberate decision, not an automatic merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Age-based retention can leave large volumes of recently downloaded MP4s on disk.
This adds
MEDIA_MAX_BYTES: after expiry and orphan cleanup, each sweep evictsthe oldest finalized MP4s until usage is within the configured budget. The default
is
0(disabled); invalid values warn and fall back to zero. For example,MEDIA_MAX_BYTES=10737418240targets a 10 GiB MP4 cache.Eviction uses the existing cross-process maintenance lock, updates media metadata
for delta sync, and preserves saved analyses and device copies. Newly published,
unreferenced files retain their one-hour grace period. Failed deletions are
attempted once per sweep while other candidates remain eligible; any remaining
excess is reported as
over_budget_bytesand logged.Operators can inspect the cache with
GET /admin/media-cache/statsand triggercleanup with
POST /admin/media-cache/sweep. Both require the existing accesstoken and run filesystem/database work off the event loop. Statistics include
file count, bytes, oldest/newest ages, budget excess, and the last completed sweep.
Sweep history persists across API workers and restarts. Unavailable operations
return 503; completed sweeps report individual deletion failures in their result.
Relative
MEDIA_PATHvalues now resolve from the backend directory so the APIand download subprocesses use the same location regardless of their working
directory. Existing deployments using a relative path from another launch
directory should set an absolute path to preserve their cache location.
This is a budget applied at each sweep. It excludes active download workspaces
and other application storage, and usage can exceed it between sweeps or when
files cannot be evicted. At 50 MB/video, 1,000 videos use about 50 GB, and
100 videos/day over 14 days use about 70 GB before cleanup. Operators should allow
temporary-download headroom and use a volume quota when an absolute cap is needed.
Configuration, monitoring requests, and capacity guidance are documented in
docs/OFFLINE_MEDIA.md, the environment example, and Docker Compose.Validation:
retention, media/admin endpoints, temporary
download cleanup, and delta-sync pagination. Includes budget boundaries,
eviction order, failed deletions, publication grace, persisted statistics,
cross-process coordination, authentication, and relative-path consistency.
checks pass.
provider downloads remain release smoke tests.