fix(seed-climate-zone-normals): proxy fallback when Open-Meteo 429s on Railway IP

[koala73]

Summary

Surfaced during PR #3097 bake on 2026-04-16: seed-climate-zone-normals failed every batch with HTTP 429 from Open-Meteo's free-tier per-IP throttle. The seeder retried with 2/4/8/16s backoff but exhausted without ever falling back to Decodo, even though the project already uses Decodo for FRED and IMF (same per-IP throttle class).

[OPEN_METEO] 429 for normals batch (Mediterranean, Taiwan Strait); retrying batch in 5s
[OPEN_METEO] 429 for normals batch (Mediterranean, Taiwan Strait); retrying batch in 10s
[OPEN_METEO] 429 for normals batch (Mediterranean, Taiwan Strait); retrying batch in 20s
[OPEN_METEO] 429 for normals batch (Mediterranean, Taiwan Strait); retrying batch in 40s
... (exhausts, throws)

This blocked the PR #3097 bake clock from starting on climate:zone-normals:v1 — the seeder couldn't write the contract envelope even when manually triggered.

Fix

After direct retries exhaust, _open-meteo-archive.mjs falls back to httpsProxyFetchRaw (Decodo) — same pattern as fredFetchJson and imfFetchJson in _seed-utils.mjs.

• Skips silently if no proxy is configured (preserves existing behavior in non-Railway envs / test runs).
• Logs [OPEN_METEO] direct exhausted on X; trying proxy so the fallback is visible in Railway logs.
• Logs [OPEN_METEO] proxy succeeded for X on success and proxy fallback failed for X: <msg> on proxy failure.
• Non-retryable statuses (500, 502 etc.) now break out of the direct-retry loop instead of throwing immediately, so they too get the proxy attempt before final exhaustion.

Test plan

• tests/open-meteo-proxy-fallback.test.mjs — 4 new cases (429-no-proxy preserves pre-fix throw, 200 OK passthrough, batch size mismatch detection, non-retryable status flow)
• npm run test:data — 5359/5359 pass (+4 new)
• node --check scripts/_open-meteo-archive.mjs — clean
• Post-merge: re-trigger seed-bundle-climate in Railway. Expect [OPEN_METEO] direct exhausted on X; trying proxy lines if Open-Meteo still throttles, then proxy succeeded + completed seed run.

Follow-up

Per PR #3097 bake plan: once climate:zone-normals:v1 writes the envelope, /api/seed-contract-probe flips to ok: true and the formal 7-day bake clock starts.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview, Comment	Apr 16, 2026 4:27am

[greptile-apps]

Greptile Summary

Adds a Decodo proxy fallback to fetchOpenMeteoArchiveBatch after direct retries exhaust, using the same httpsProxyFetchRaw pattern already in place for FRED and IMF. The fix correctly converts the immediate throw to a break for both retryable-status-exhaustion and non-retryable statuses, then attempts the proxy before the final throw.

Two minor gaps worth noting:

• The catch block for network errors (timeouts, connection resets) still contains a bare throw err on the final attempt, which exits the function without reaching the proxy block — inconsistent with how HTTP-status failures are handled and how imfFetchJson approaches the same pattern.
• The final thrown error no longer carries the HTTP status code, reducing signal in Railway logs and monitoring.

Confidence Score: 5/5

Safe to merge — the primary 429-exhaustion bug is correctly fixed and all remaining findings are P2 style/robustness suggestions.

The fix targets the specific failure mode (HTTP 429 exhausting all direct retries with no proxy fallback) and the logic is sound. The two flagged items — network errors bypassing the proxy and the loss of HTTP status in the final error message — are pre-existing limitations or minor observability concerns that don't affect correctness.

scripts/_open-meteo-archive.mjs — specifically the catch block's final throw err path (lines 63–71) which still bypasses the proxy.

Important Files Changed

Filename	Overview
scripts/_open-meteo-archive.mjs	Replaces immediate throw with break+proxy-fallback pattern; core 429-retry logic is sound, but network-error path in the catch block still throws directly and bypasses the proxy.
tests/open-meteo-proxy-fallback.test.mjs	Four new tests covering the no-proxy-throw, 200-passthrough, batch-mismatch, and non-retryable-break paths; proxy integration deferred to post-merge Railway validation with clear acknowledgement in comments.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([fetchOpenMeteoArchiveBatch called]) --> B[Build URL with query params]
    B --> C{attempt <= maxRetries?}
    C -- No --> G
    C -- Yes --> D[fetch URL direct]
    D -- network error --> E{attempt < maxRetries?}
    E -- Yes --> F[sleep backoff] --> C
    E -- No --> THROW_NET([throw err ❌ bypasses proxy])
    D -- resp.ok --> OK[parse + validate batch size]
    OK -- mismatch --> THROW_MISMATCH([throw batch size mismatch])
    OK -- ok --> RETURN([return data ✅])
    D -- non-2xx --> H{retryable AND attempt < maxRetries?}
    H -- Yes --> F
    H -- No --> G[break out of loop]
    G --> I{proxyAuth configured?}
    I -- No --> THROW_EX([throw retries exhausted])
    I -- Yes --> J[log direct exhausted; trying proxy]
    J --> K[httpsProxyFetchRaw via Decodo]
    K -- ok --> L[parse + validate batch size]
    L -- mismatch --> M[catch: log proxy failed]
    L -- ok --> N([log proxy succeeded; return data ✅])
    K -- error --> M
    M --> THROW_EX

Loading

Comments Outside Diff (1)

1. scripts/_open-meteo-archive.mjs, line 63-71 (link)

   Network errors on final retry bypass proxy fallback

   When fetch throws (timeout, ECONNRESET, DNS failure) on the last attempt, the throw err inside the catch block exits the loop and propagates directly, skipping the proxy fallback block below. If Railway IPs were ever TCP-blocked by Open-Meteo rather than receiving HTTP 429s, no proxy attempt would be made despite one being configured.

   The peer helper imfFetchJson avoids this gap by wrapping both network errors and non-2xx responses in a single try/catch that unconditionally falls through to the proxy. Converting the catch path here to break when attempt >= maxRetries would give network errors the same proxy opportunity as HTTP errors:

     } catch (err) {
       if (attempt < maxRetries) {
         const retryMs = retryBaseMs * 2 ** attempt;
         console.log(`  [OPEN_METEO] ${err?.message ?? err} for ${label}; retrying batch in ${Math.round(retryMs / 1000)}s`);
         await sleep(retryMs);
         continue;
       }
       // Final attempt: fall through to proxy, same as HTTP error path
       break;
     }

_{Reviews (1): Last reviewed commit: "fix(seed-climate-zone-normals): proxy fa..." | Re-trigger Greptile}

[greptile-apps]

Final throw drops HTTP status from error message

The original error included the HTTP status (Open-Meteo ${resp.status} for ${label}), which is visible in Railway log tails and in any monitoring system that ingests thrown error messages. The replacement message retries exhausted omits the last-seen status code; the only place the status appears is in the per-retry console.warn lines, which may scroll off or not be correlated. Embedding the last-seen status (tracked as a variable across the loop) in the final throw would preserve the diagnostics without extra complexity.

→ consider tracking lastStatus in the loop body and throwing Open-Meteo retries exhausted (last status ${lastStatus}) for ${label}.

[koala73]

Addressed all three review findings in ecce579:

P1 (yours) — throw err in catch block bypassed proxy fallback for thrown errors (timeout, ECONNRESET, DNS). Fixed: catch now records lastDirectError and breaks, so control falls through to the proxy fallback regardless of whether the direct path failed via thrown error or non-OK status.

P2 (yours) — tests didn't exercise the proxy path. Refactored helper to accept _proxyResolver and _proxyFetcher opt overrides (production defaults to real resolveProxy/httpsProxyFetchRaw from _seed-utils.mjs; tests inject mocks). Added 4 new cases covering: proxy success returns data, thrown-error → proxy runs (P1 regression guard with explicit directCalls=2, proxyCalls=1 assertion), proxy ALSO fails → throws exhausted with original error in cause, proxy returns wrong batch size → caught + warns + throws.

P2 (Greptile) — final throw dropping HTTP status. Final message now includes (last direct: ${lastDirectError.message}) which is HTTP <status> for status-code failures and the original error message for thrown errors, plus Error.cause for chained inspection. Same diagnostic info as before, in a more general form that also covers the thrown-error path.

Tests: 8/8 (was 4); test:data 5363/5363 (+4); typecheck clean.