Skip to content

Extend the radar loop from NOAA's 50 minutes to a configurable 24 hours - #35

Merged
turbo5000c merged 7 commits into
mainfrom
claude/radar-loop-duration-l7auzx
Aug 23, 2026
Merged

Extend the radar loop from NOAA's 50 minutes to a configurable 24 hours#35
turbo5000c merged 7 commits into
mainfrom
claude/radar-loop-duration-l7auzx

Conversation

@turbo5000c

@turbo5000c turbo5000c commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Description

The Radar Loop image entity showed NOAA's ready-made animation, which is fixed at ten frames covering roughly 50 minutes. Only those ten frames exist on NOAA's server, so a longer loop cannot be downloaded — it has to be collected. This PR makes the entity collect one frame per refresh, file each under the time NOAA published it, and assemble the animation itself from an evenly time-spaced sample.

The window is a new options-flow step, defaulting to 24 hours. Setting it to 0 restores the previous behaviour exactly: NOAA's own loop, proxied unchanged, with nothing written to disk.

How it behaves

  • The loop fills in over time. Below six frames the card shows NOAA's own loop instead, so it is never blank and never worse than before.
  • Frames survive restarts, stored one small GIF per scan under <config>/noaa_it_all/radar_frames/<RADAR_SITE>/ — a few megabytes per radar site. Frames outside the window are pruned on every refresh. The directory is removed when the integration is deleted, when the entry switches to another office, and when the option is set back to 0 — unless another configured office still builds a loop from the same site.
  • A 24-hour loop is capped at 72 frames (one every 20 minutes) and plays through in about ten seconds. Shorter windows are proportionally finer — a 6-hour loop keeps roughly one frame per scan.
  • loop_mode reports which animation is on screen (local or upstream), alongside loop_hours, frame_count, window_start and window_end, so a loop quietly shorter than configured is visible from a template.

Two things the encoder has to get right, both pinned by tests that fail against the obvious implementation:

  • Frames arrive separately palettised. Letting the GIF writer adopt frame one's palette as the global table remaps everything after it. Building the palette from frame one alone is wrong in a way that only shows up overnight: the loop starts on an empty radar, so the palette holds nothing but background, and the storm arriving hours later renders as background too. The palette is sampled across the whole window instead.
  • Pillow merges pixel-identical neighbouring frames and sums their delays, so a clear night legitimately encodes to fewer frames than it has sources. Playback time and the hold on the newest frame both survive that, so it stands as free compression.

Behaviour changes worth calling out

  • The locally built loop is opaque where NOAA's is transparent. Frames are composited onto solid black before being combined — reconciling per-frame transparency across differing palettes is the most reliable way to produce a psychedelic radar loop. Cards relying on transparency should set the option to 0.
  • Cached HTTP validators are now kept per upstream resource. This entity fetches two URLs, and an ETag from the single frame must never be sent as a validator for the animation.

Not included: seeding the buffer from NOAA's ten-frame loop at first start. Its per-frame cadence is unpublished and varies by scan strategy, so every back-dated timestamp would be a guess polluting an otherwise exactly-timestamped buffer — for 50 minutes of history on a 24-hour feature that the upstream fallback already covers.

Review pass

A pre-merge review found fifteen issues, all fixed in d83fa39. Each fix is pinned by a test that fails without it.

The worst was a time bomb in this PR's own tests: the two end-to-end radar tests pinned frame timestamps to a calendar date while the code prunes against the wall clock, so from the next day every frame would have fallen outside the window and CI would have gone red with no code change. Timestamps now derive from the clock; verified by running the suite with the clock 2, 40 and 400 days ahead.

Correctness bugs that would otherwise have shipped:

  • select_frames applied its window only after the "few enough frames, take them all" shortcut, so a buffer pruning had not caught up with built a loop from the wrong day and reported it as the configured window.
  • Pruning ran only when a new frame was stored, so a radar site stuck on one scan for hours never pruned at all.
  • The prune had no upper bound on frame age. A future-dated frame from a wrong clock is never reached by a backwards-looking cutoff, so it would outlive every real frame. The file cap did not help — it trims the oldest, and those frames sort newest.
  • The oversize retry thinned with paths[::2], which drops the last frame whenever the count is even — the normal case at 72 — so the animation ended, and held, on a stale scan.
  • The RADAR_LOOP_MIN_FRAMES floor was checked against every frame on disk rather than the sampled subset that actually ships, so a window with a long outage could send a four-frame loop in place of NOAA's ten.
  • The canonical frame size came from the oldest frame, so after a NOAA product-size change every new full-resolution scan would be scaled back down for a whole window.

Regressions introduced earlier in this PR and fixed in the same commit:

  • The single validator slot meant the frame fetch and the fallback fetch evicted each other's ETags every cycle, making 304s structurally unreachable for the fallback rather than merely safe.
  • Passing an explicit URL skipped cache busting, so the fallback loop lost the ?t= it had before this PR.

Plus: frame_count/window_end reported frames offered to the encoder rather than used — hiding exactly the shortfall those attributes exist to reveal; switching office or setting hours to 0 orphaned the old site's frames forever, contradicting the options screen's "store nothing"; and the invalid-hours error hardcoded "between 0 and 24" while the step description interpolates the constant.

Type

  • Bugfix
  • Feature
  • Documentation
  • Other

Checklist

  • Follows Home Assistant custom integration structure
  • Passes flake8 checks (max-line-length=120)
  • Manifest is valid and complete
  • README updated if required

Verification

  • 850 tests pass (up from 764), flake8 clean, all JSON valid, strings.json and translations/en.json byte-identical. Pillow added to the CI install line so the encoder tests actually run there rather than skipping.
  • Pillow is not declared in manifest.json — it is already a Home Assistant core requirement, and the module degrades to NOAA's loop if the import fails.
  • Mutation-tested: each of the ten fixes above was reintroduced deliberately and confirmed to turn a test red. One test that passed under mutation was rewritten until it caught the defect it named.
  • Clock-shifted: the full suite passes with the wall clock moved 2, 40 and 400 days forward.
  • End-to-end against a real directory and a real Pillow: 144 frames of simulated polling sample to 72 at 20-minute spacing, encode in 0.35 s, produce a 191 KiB GIF that plays for exactly 10.0 s with the reflectivity scale intact and the newest scan last.

Known follow-ups (not blocking)

  • {SITE}_0.gif is fetched twice per cycle — once by this entity, once by the base-reflectivity entity — plus a HEAD from the radar timestamp coordinator. Deduplicating means either coupling the two entities or adding a shared per-site fetch cache.
  • The encoder rebuilds from scratch on every refresh (~151 file opens, ~79 decodes) to incorporate one new scan. Measured at 0.35 s per 10 minutes, so affordable; caching the quantised palette would cut it by roughly an order of magnitude.

claude added 7 commits August 23, 2026 19:56
The radar loop is about to display a GIF it assembles from many fetches
rather than the bytes of any single one, which breaks the assumption that
the cached picture and the last HTTP response are the same object.
Conditional revalidation and the 304 short-circuit now key off a
dedicated _last_fetched_bytes, _async_fetch_image() takes an optional URL
so a fallback path need not reimplement its error handling, and the
recovery bookkeeping moves into _note_success() so a subclass can reuse
it.

No behaviour change: for every entity that exists today the two byte
caches move in lockstep.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FticTYLC8japDARGL6mcLH
NOAA keeps only ten frames on the server, so a loop longer than its own
fifty minutes cannot be downloaded -- it has to be accumulated. This adds
the machinery: a per-radar-site directory of single-scan GIFs named by
scan time, a sampler that picks evenly spaced frames by time rather than
by position, and a Pillow encoder.

Two things the encoder has to get right, both covered by tests that fail
against the obvious implementation:

Frames arrive separately palettised, and letting the GIF writer adopt the
first frame's palette as the global table remaps every later frame
against it. A palette built from frame one alone is just as wrong in a
way that only shows up overnight: the loop starts on an empty radar, so
the palette holds nothing but background, and the storm that arrives
hours later is rendered as background too. The palette is therefore
sampled across the whole window.

Pillow also merges pixel-identical neighbouring frames and sums their
delays, so a clear night legitimately encodes to fewer frames than it has
sources. Playback time and the hold on the newest frame both survive
that, so it stands as free compression.

Nothing in the module imports Home Assistant, which is what lets its
tests run against a real directory and a real Pillow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FticTYLC8japDARGL6mcLH
A third options step asks how many hours of radar the loop should cover,
defaulting to a full day and accepting 0 to keep serving NOAA's own
fifty-minute animation and store nothing.

The step writes all four option keys explicitly. Options replace the
stored mapping rather than merging into it, so a key the final step
forgets is a key silently dropped the next time anyone opens Configure --
there is a regression test pinning that down. The bounds are checked in
Python as well as in the schema because the flow tests mock voluptuous
wholesale, which would make a vol.Range accept anything.

Initial setup is left at two steps: the loop writes frames into the
configuration directory, and an upgrade should not start doing that to
everyone who never opened the options page.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FticTYLC8japDARGL6mcLH
The Radar Loop entity now polls the latest single scan rather than NOAA's
finished animation, files each new one under its published time, and
assembles the loop itself from a time-spaced sample of what it holds. The
loop starts short after a fresh install and fills out over the following
day; frames outlive restarts, so that happens once rather than on every
reboot.

Frames are identified by Last-Modified, which is when NOAA published the
scan. That puts them on the real volume-scan cadence instead of our
refresh boundary, and makes two refreshes that see the same scan resolve
to the same file. Hashing the bytes would have been actively wrong: two
scans of a clear sky are genuinely identical, so a quiet night would
collapse to one frame and the loop would cut straight from clear to storm
with no sense of time passing.

Validators are now offered back only to the resource they came from. This
entity fetches two URLs, and an ETag from the single frame must never be
offered as a validator for the animation -- a server answering 304 to
that would hand back the wrong image.

Everything degrades to NOAA's own loop: too few frames yet, Pillow
missing, assembly failing, a disk that will not take the frame. Every
failure path leaves the displayed animation exactly as it was, and the
loop_mode attribute reports which animation is actually on screen rather
than which one was configured.

Removing the integration deletes the frames, unless another entry shares
the radar site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FticTYLC8japDARGL6mcLH
Covers what the option does, that the loop fills in over time, where the
frames live and roughly what they cost, and the two behaviour changes an
existing user would otherwise be surprised by: the locally built loop is
opaque where NOAA's is transparent, and 0 restores the old behaviour
exactly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FticTYLC8japDARGL6mcLH
The existing radar tests swap either the store or the encoder for a fake,
which leaves the wiring between them untested -- and that wiring is where
a 24-hour loop either works or quietly serves NOAA's fifty minutes
forever. Two tests now drive the real entity over a real directory with a
real Pillow: one watches the loop grow from NOAA's animation into a
locally built one, the other restarts the entity over the same directory
and checks the history is still there.

Also pins the rule that cached validators are only offered back to the
URL they came from, in both directions, plus the case it must not break:
an entity with a single URL still revalidates.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FticTYLC8japDARGL6mcLH
A pre-merge review turned up fifteen issues; each fix below is pinned by a
test that fails without it.

Worst first, a time bomb in my own tests: the two end-to-end radar tests
pinned their frame timestamps to a calendar date while the code prunes
against the wall clock, so every frame would have fallen outside the
window from the next day onward and CI would have gone red with no code
change. Timestamps are now derived from the clock. Verified by running
the suite with the clock two days, forty days and a year ahead.

Correctness:
- select_frames applied its window only after the "few enough frames,
  take them all" shortcut, so a buffer pruning had not caught up with
  built a loop out of the wrong day and then reported it as the
  configured window.
- Pruning ran only when a new frame was stored. A radar site stuck on one
  scan for hours -- maintenance, an outage -- never pruned at all, which
  is also what the docs already claimed did not happen.
- The prune had no upper bound on frame age. A frame dated into the
  future by a wrong clock is never reached by the cutoff, so it would
  outlive every real frame and strand the loop on a set that never
  updates. The file cap did not help: it trims the oldest, and those
  frames sort newest.
- The oversize retry thinned with paths[::2], which drops the last frame
  whenever the count is even -- the normal case at 72. The animation
  ended, and held, on a scan at least twenty minutes stale.
- The RADAR_LOOP_MIN_FRAMES floor was checked against every frame on disk
  but the loop is built from the sampled subset, so a window with a long
  outage could ship a four-frame loop in place of NOAA's ten.
- The newest frame now sets the canonical size. Taking it from the oldest
  meant that after a NOAA product-size change, every new full-resolution
  scan was scaled back down to the superseded dimensions for a whole
  window.

Regressions this PR had introduced:
- The single validator slot meant the frame fetch and the fallback loop
  fetch evicted each other's ETags every cycle, so the fallback could
  never revalidate -- the scoping fix made 304s structurally unreachable
  rather than merely safe. Validators are now kept per resource.
- Passing an explicit URL skipped cache busting, so the fallback loop
  lost the ?t= it had always had and could be served a stale CDN copy.

Honesty of the exposed attributes:
- frame_count and window_end came from the frames offered to the encoder
  rather than the ones it used, hiding exactly the shortfall the
  attributes exist to reveal. assemble_gif now returns the frames it
  encoded alongside the bytes.

Housekeeping:
- Switching forecast office, or setting the duration back to 0, orphaned
  the old radar site's frames forever: async_remove_entry only knows the
  site the entry ends on, and a store that is never constructed never
  prunes. Setup now sweeps directories no configured entry is collecting
  for, which is what makes the options screen's "store nothing" true.
- The invalid-hours error hardcoded "between 0 and 24" while the step
  description interpolates the constant.

Also replaces a sampler test that never reached the sampler -- its
16-frame fixture hit the take-them-all shortcut, so the tolerance logic it
named could have been deleted without failing it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FticTYLC8japDARGL6mcLH
@turbo5000c
turbo5000c marked this pull request as ready for review August 23, 2026 21:15
@turbo5000c
turbo5000c merged commit 240dc9a into main Aug 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants