Keep NOAA image cards on screen through network blips

CHANGELOG.md

@@ -5,7 +5,102 @@ All notable changes to NOAA It All for Home Assistant will be documented in this
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.5.2] - Current
+## [0.5.3] - Current
+
+### Fixed
+- **A network blip no longer blanks the NOAA image cards.** Every image entity's `async_image()`
+  returned `b""` on any failure, and Home Assistant treats empty bytes as an error and turns them
+  into an HTTP 500 — so a momentary `Cannot connect to host services.swpc.noaa.gov:443 ... [Timeout
+  while contacting DNS servers]` was enough to replace a perfectly good picture with a broken tile.
+  Nothing was cached, so there was nothing to fall back on. The image bytes are now kept in memory
+  and re-served: a failed refresh changes neither the cached frame nor `image_last_updated`, so the
+  previous picture stays on the dashboard until a later refresh replaces it.
+- **Images are fetched on a timer instead of while serving the HTTP request.** All seven entities
+  now fetch in the background every 10 minutes, so a slow NOAA can no longer blow Home Assistant's
+  10-second image-proxy budget, and several dashboard clients asking at once no longer each start
+  their own request. The first fetch is scheduled rather than awaited during setup, so an
+  unreachable NOAA cannot hold up the config entry.
+- **`entity_picture` now points at Home Assistant's image proxy** (`/api/image_proxy/...`) once a
+  frame has been fetched, instead of always sending the browser straight to `services.swpc.noaa.gov`.
+  This is what makes the cache reachable — previously the browser fetched NOAA itself and the
+  entity's own bytes were never used, so the card broke whenever *the browser* could not reach NOAA.
+  Until the first successful fetch the entity still falls back to the upstream URL, so a restart
+  while Home Assistant's own resolver is broken still renders if the browser's network is fine.
+- **A total-request timeout is no longer reported as an unexpected error.** `aiohttp`'s
+  `ClientTimeout` expiry raises `asyncio.TimeoutError`, which is not an `aiohttp.ClientError`, so it
+  fell through to the catch-all arm and logged `Unexpected error fetching ... image`. Timeouts, DNS
+  failures, connection resets and server disconnects are now classified together as transient.
+- **Transient failures no longer log an error per blip.** A `cloud_polling` integration losing its
+  upstream for a minute is normal. Consecutive failures now stay at debug while a cached frame is
+  still being served, warn once the outage has lasted about half an hour, and only escalate to error
+  after roughly an hour — and then only periodically. Recovery logs a single info line. A failure
+  that is *not* transient (a 404, a content type that is not an image) still warns immediately, as
+  does any failure while there is no cached image to show.
+- **The declared content type now matches the actual image format.** Home Assistant defaults every
+  image entity to `image/jpeg`; five of the seven are not JPEGs. The geoelectric field and hurricane
+  outlook images are PNG, both radar images are GIF, and the content type reported by NOAA is
+  adopted when it differs.
+- **A single failed Points API lookup no longer disables forecasts until a restart.** This is the
+  cause of the recurring `Error fetching NOAA Forecasts data: All forecast API requests failed`.
+  `ForecastCoordinator._resolve_forecast_urls()` set `self._urls_fetched = True` in its `except`
+  branch as well as on success, so one transient failure left both forecast URLs `None` with no way
+  to retry — and because each fetch is guarded by `if self._forecast_url:`, no request was even
+  attempted afterwards. Every subsequent refresh went straight to `UpdateFailed`, forever. The flag
+  is now only latched on success, so the next 10-minute cycle re-resolves. The same latch was in
+  `ObservationsCoordinator._resolve_station()` and `CloudCoverCoordinator._resolve_gridpoint_url()`,
+  where it silently retired the observation-station and gridpoint lookups; both are fixed too.
+- **Space weather and hurricane requests now send a `User-Agent`.** They were the only 5 of 19
+  outbound requests without one, and `_HURRICANE_ALERTS_URL` points at `api.weather.gov`, which
+  requires it.
+- **`All X API requests failed` now says which endpoints failed and why.** The message discarded
+  every underlying exception, so the log line naming the problem was useless on its own and the real
+  cause sat in separate `WARNING` lines above it — when a request had been attempted at all. Failures
+  are now collected and appended, e.g. `All forecast API requests failed: Points API lookup
+  (ClientConnectorError: Cannot connect to host api.weather.gov:443 ...)`.
+- **`coordinator.py` now has behavioural tests** (`tests/test_coordinator.py`). It had none, across
+  773 lines and 10 coordinators, which is how the latch bug survived. Every new test was confirmed
+  to fail against the pre-fix code.
+
+### Changed
+- **The seven image entity classes now share a `NoaaImageEntity` base.** Each was a near-identical
+  copy of the same ~70 lines, which is why the `b""` bug existed in seven places at once. Subclasses
+  keep only what differs: name, unique ID, device info, upstream URL, content type and a log label.
+- **Image entities now report a state.** Previously all seven sat at `unknown` forever, because
+  `image_last_updated` was never set. The state is now an ISO-8601 timestamp that advances whenever
+  the image bytes change, which also makes "this image has gone stale" templatable.
+- **The `User-Agent` now identifies this integration honestly.** It was
+  `HomeAssistant/NOAA-Integration` on all 17 outbound request sites: generic, unversioned, carrying no
+  contact information, and implying Home Assistant core rather than a third-party custom integration.
+  `api.weather.gov` requires a User-Agent and asks that it be unique to the application, with a website
+  or email so they can make contact instead of simply blocking traffic they cannot place — which matters
+  more now that the integration polls on a timer. It is now
+  `noaa_it_all/<version> (+https://github.com/dawg-io/noaa_it_all)`, built from `manifest.json` at import
+  so a release bump is the only edit needed — `const.VERSION` and `const.DOCUMENTATION_URL` now read from
+  there, and `tests/test_manifest.py` fails if either is ever pasted back in as a literal. A contact email
+  may be added to the string later.
+- **Refreshes revalidate with `ETag` / `Last-Modified`.** Because the integration now polls whether
+  or not anyone is looking at the dashboard, conditional requests keep the steady-state cost close
+  to zero for sources that publish infrequently. Requests also send the integration's `User-Agent`,
+  matching the coordinators.
+
+- **The documented dashboard templates no longer blow up during startup.** Home Assistant renders
+  dashboard templates as soon as the frontend subscribes to them, which on a cold boot can be before
+  this integration has registered its entities — `async_setup_entry` awaits an initial refresh of ten
+  coordinators, all making live NWS calls, before forwarding any platform. In that window
+  `state_attr(...)` and `states.sensor....` both return `None`, so the README's own examples raised
+  `TypeError: 'NoneType' object is not iterable` and `UndefinedError: None has no element 0`, one
+  traceback per card, intermittently. The three Extended Forecast cards, the upcoming-meteor-shower
+  loop, the mobile header and the two alert automations now guard with `or []` and a truth test, and
+  the Dashboard Card Examples section explains the race so new cards get written the same way. The
+  sensors themselves were never at fault: `periods` and `upcoming` are always published as lists.
+- **Removed a duplicated "Dashboard Card Examples" heading** in `README.md`.
+
+### Known limitations
+- Two configured NWS offices means two entities fetching the byte-identical geoelectric and aurora
+  images, since those URLs are office-independent. Harmless but wasteful; a shared per-URL fetcher
+  is the follow-up.
+
+## [0.5.2]
 
 ### Fixed
 - **Image entities no longer log an error on every startup.** `image.py` sets up its entities with

README.md

@@ -268,7 +268,9 @@ Visual representations of current conditions:
 - **Radar Base Reflectivity** — Latest NEXRAD base reflectivity radar for your NWS office *(image.noaa_{office}_weather_radar_base_reflectivity)*
 - **Radar Loop** — Animated NEXRAD radar loop *(image.noaa_{office}_weather_radar_loop)*
 
-> **Tip**: Image entities can be displayed on dashboards using the standard `picture-entity` or `picture-glance` cards. Note that image entities are not polled by Home Assistant: their URL is resolved once when the integration loads, so the picture refreshes only when the browser re-fetches it or the config entry is reloaded.
+> **Tip**: Image entities can be displayed on dashboards using the standard `picture-entity` or `picture-glance` cards.
+>
+> Home Assistant fetches each image from NOAA in the background every 10 minutes and serves it to the browser through its own image proxy (`/api/image_proxy/...`), so the dashboard does not talk to NOAA directly. The most recently fetched frame is kept in memory: if NOAA is briefly unreachable the card keeps showing the last good image rather than going blank, and the outage is logged quietly unless it persists. Each entity's state is the timestamp of the image currently being served, so you can alert on an image going stale.
 
 ## NWS Forecast Offices
 
@@ -442,15 +444,18 @@ automation:
     action:
       - service: notify.mobile_app_your_phone
         data:
-          title: "⚠️ {{ state_attr('binary_sensor.noaa_ilm_weather_active_alerts','alerts')[0].event }}"
-          message: "{{ state_attr('binary_sensor.noaa_ilm_weather_active_alerts','alerts')[0].description | replace('\r\n',' ') }}"
+          title: >-
+            ⚠️ {{ (state_attr('binary_sensor.noaa_ilm_weather_active_alerts','alerts') or [{}])[0].get('event', 'Weather alert') }}
+          message: >-
+            {{ (state_attr('binary_sensor.noaa_ilm_weather_active_alerts','alerts') or [{}])[0].get('description', 'See the NWS alert for details.') | replace('\r\n',' ') }}
           data:
             priority: high
             notification_icon: mdi:weather-lightning-rainy
       - service: tts.google_translate_say
         data:
           entity_id: media_player.home_speaker
-          message: "{{ state_attr('binary_sensor.noaa_ilm_weather_active_alerts','alerts')[0].description | replace('\r\n',' ') }}"
+          message: >-
+            {{ (state_attr('binary_sensor.noaa_ilm_weather_active_alerts','alerts') or [{}])[0].get('description', 'See the NWS alert for details.') | replace('\r\n',' ') }}
 ```
 
 #### Winter Storm Alert with Light Flash
@@ -758,10 +763,28 @@ script:
 
 ### Dashboard Card Examples
 
-### Dashboard Card Examples
-
 These examples demonstrate how to create effective dashboard cards organized by device groups.
 
+> **Guard against startup, or your log will fill with tracebacks.** Home Assistant renders
+> dashboard templates as soon as the frontend subscribes to them, which on a cold boot can be
+> *before* this integration has registered its entities — the config entry awaits an initial
+> refresh of ten coordinators, all making live NWS calls, before any platform is set up. Until
+> that finishes, `state_attr('sensor.noaa_…', 'periods')` returns `None` and
+> `states.sensor.noaa_…` returns `None`, so an unguarded `[0]` or `{% for %}` raises
+> `TypeError: 'NoneType' object is not iterable` or `UndefinedError: None has no element 0`.
+> The card recovers on the next render, but each attempt logs a full traceback, and it is
+> intermittent — it depends on whether the templates lose the race that boot.
+>
+> Two habits avoid all of it:
+>
+> - **List attributes**: `{% set items = state_attr(…, 'periods') or [] %}`, then check
+>   `{% if items | count > 0 %}` before indexing. The integration itself always publishes these
+>   as lists, never `None` — a `None` means the entity is not there yet.
+> - **`states.` objects**: bind first and test for truth —
+>   `{% set s = states.sensor.noaa_… %}{{ … if s else '—' }}`.
+>
+> The examples below all do this.
+
 #### Weather Alerts Card (NOAA Weather [OFFICE] Group)
 ```yaml
 type: entities
@@ -812,53 +835,53 @@ entities:
 type: vertical-stack
 cards:
   - type: markdown
-    content: >
-      <div style="display:flex;align-items:center;gap:14px;"> <img src="{{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[0].icon
-      }}" width="56"> <div> <b>{{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[0].name
-      }} · {{
-      as_timestamp(state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[0].start_time)
-      | timestamp_custom('%a %m/%d') }}</b><br> <span
-      style="font-size:1.4em;font-weight:700;"> {{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[0].temperature
-      }}°{{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[0].temperature_unit
-      }} </span><br> {{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[0].detailed_forecast
-      }} </div></div>
+    content: |
+      {% set periods = state_attr('sensor.noaa_ilm_weather_extended_forecast','periods') or [] %}
+      {% if periods | count > 0 %}
+      {% set p = periods[0] %}
+      <div style="display:flex;align-items:center;gap:14px;">
+      <img src="{{ p.icon }}" width="56">
+      <div>
+      <b>{{ p.name }} · {{ as_timestamp(p.start_time) | timestamp_custom('%a %m/%d') }}</b><br>
+      <span style="font-size:1.4em;font-weight:700;">{{ p.temperature }}°{{ p.temperature_unit }}</span><br>
+      {{ p.detailed_forecast }}
+      </div>
+      </div>
+      {% else %}
+      _Forecast loading…_
+      {% endif %}
   - type: markdown
-    content: >
-      <div style="display:flex;align-items:center;gap:14px;"> <img src="{{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[1].icon
-      }}" width="56"> <div> <b>{{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[1].name
-      }} · {{
-      as_timestamp(state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[1].start_time)
-      | timestamp_custom('%a %m/%d') }}</b><br> <span
-      style="font-size:1.4em;font-weight:700;"> {{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[1].temperature
-      }}°{{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[1].temperature_unit
-      }} </span><br> {{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[1].detailed_forecast
-      }} </div></div>
+    content: |
+      {% set periods = state_attr('sensor.noaa_ilm_weather_extended_forecast','periods') or [] %}
+      {% if periods | count > 1 %}
+      {% set p = periods[1] %}
+      <div style="display:flex;align-items:center;gap:14px;">
+      <img src="{{ p.icon }}" width="56">
+      <div>
+      <b>{{ p.name }} · {{ as_timestamp(p.start_time) | timestamp_custom('%a %m/%d') }}</b><br>
+      <span style="font-size:1.4em;font-weight:700;">{{ p.temperature }}°{{ p.temperature_unit }}</span><br>
+      {{ p.detailed_forecast }}
+      </div>
+      </div>
+      {% else %}
+      _Forecast loading…_
+      {% endif %}
   - type: markdown
-    content: >
-      <div style="display:flex;align-items:center;gap:14px;"> <img src="{{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[2].icon
-      }}" width="56"> <div> <b>{{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[2].name
-      }} · {{
-      as_timestamp(state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[2].start_time)
-      | timestamp_custom('%a %m/%d') }}</b><br> <span
-      style="font-size:1.4em;font-weight:700;"> {{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[2].temperature
-      }}°{{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[2].temperature_unit
-      }} </span><br> {{
-      state_attr('sensor.noaa_ilm_weather_extended_forecast','periods')[2].detailed_forecast
-      }} </div></div>
+    content: |
+      {% set periods = state_attr('sensor.noaa_ilm_weather_extended_forecast','periods') or [] %}
+      {% if periods | count > 2 %}
+      {% set p = periods[2] %}
+      <div style="display:flex;align-items:center;gap:14px;">
+      <img src="{{ p.icon }}" width="56">
+      <div>
+      <b>{{ p.name }} · {{ as_timestamp(p.start_time) | timestamp_custom('%a %m/%d') }}</b><br>
+      <span style="font-size:1.4em;font-weight:700;">{{ p.temperature }}°{{ p.temperature_unit }}</span><br>
+      {{ p.detailed_forecast }}
+      </div>
+      </div>
+      {% else %}
+      _Forecast loading…_
+      {% endif %}
 ```
 
 #### Meteor Shower Card (NOAA Space Group)
@@ -914,7 +937,8 @@ To list the upcoming showers, read the `upcoming` attribute with a markdown card
 type: markdown
 title: ☄️ Upcoming Meteor Showers
 content: |
-  {% for s in state_attr('sensor.noaa_ilm_space_next_meteor_shower', 'upcoming') %}
+  {% set showers = state_attr('sensor.noaa_ilm_space_next_meteor_shower', 'upcoming') or [] %}
+  {% for s in showers %}
   **{{ s.name }}** — {{ s.peak_local | as_timestamp | timestamp_custom('%b %-d') }}
   ({{ s.days_until | round(0) | int }} days), up to {{ s.zhr_max }}/hr in {{ s.constellation }}
   {% endfor %}
@@ -1094,7 +1118,8 @@ cards:
   - type: markdown
     content: |
       # 📍 Wilmington Weather
-      Updated: {{ as_timestamp(states.sensor.noaa_ilm_weather_temperature.last_changed) | timestamp_custom('%I:%M %p') }}
+      {% set t = states.sensor.noaa_ilm_weather_temperature %}
+      Updated: {{ as_timestamp(t.last_changed) | timestamp_custom('%I:%M %p') if t else '—' }}
 
   - type: entities
     entities: