fix(seed-climate-zone-normals): proxy fallback when Open-Meteo 429s on Railway IP (#3118)

* fix(seed-climate-zone-normals): proxy fallback when Open-Meteo 429s on Railway IP

Railway logs.1776312819911.log showed seed-climate-zone-normals failing
every batch with HTTP 429 from Open-Meteo's free-tier per-IP throttle
(2026-04-16). The seeder retried with 2/4/8/16s backoff but exhausted
without ever falling back to the project's Decodo proxy infrastructure
that other rate-limited sources (FRED, IMF) already use.

Open-Meteo throttles by source IP. Railway containers share IP pools and
get 429 storms whenever zone-normals fires (monthly cron — high churn
when it runs). Result: PR #3097's bake clock for climate:zone-normals:v1
couldn't start, because the seeder couldn't write the contract envelope
even when manually triggered.

Fix: after direct retries exhaust, _open-meteo-archive.mjs falls back to
httpsProxyFetchRaw (Decodo) — same pattern as fredFetchJson and
imfFetchJson in _seed-utils.mjs. Skips silently if no proxy is configured
(preserves existing behavior in non-Railway envs).

Added tests/open-meteo-proxy-fallback.test.mjs (4 cases):
- 429 with no proxy → throws after exhausting retries (pre-fix behavior preserved)
- 200 OK → returns parsed batch without touching proxy path
- batch size mismatch → throws even on 200
- Non-retryable 500 → break out, attempt proxy, throw exhausted (no extra
  direct retry — matches new control flow)

Verification: npm run test:data → 5359/5359, +4 new. node --check clean.

Same pattern can be applied to any other helper that fetches Open-Meteo
(grep 'open-meteo' scripts/) if more 429s show up.

* fix: proxy fallback runs on thrown direct errors + actually-exercised tests

Addresses two PR #3118 review findings.

P1: catch block did 'throw err' on the final direct attempt, silently
bypassing the proxy fallback for thrown-error cases (timeout, ECONNRESET,
DNS failures). Only non-OK HTTP responses reached the proxy path. Fix:
record the error in lastDirectError and 'break' so control falls through
to the proxy fallback regardless of whether the direct path failed via
thrown error or non-OK status.

Also: include lastDirectError context in the final 'retries exhausted'
message + Error.cause so on-call can see what triggered the fallback
attempt (was: opaque 'retries exhausted').

P2: tests didn't exercise the actual proxy path. Refactored helper to
accept _proxyResolver and _proxyFetcher opt overrides (production
defaults to real resolveProxy/httpsProxyFetchRaw from _seed-utils.mjs;
tests inject mocks). Added 4 new cases:

- 429 + proxy succeeds → returns proxy data
- thrown fetch error on final retry → proxy fallback runs (P1 regression
  guard with explicit assertion: directCalls=2, proxyCalls=1)
- 429 + proxy ALSO fails → throws exhausted, original HTTP 429 in
  message + cause chain
- Proxy returns wrong batch size → caught + warns + throws exhausted

Verification:
- tests/open-meteo-proxy-fallback.test.mjs: 8/8 pass (4 added)
- npm run test:data: 5363/5363 pass (+4 from prior 5359)
- node --check clean

Author: koala73

scripts/_open-meteo-archive.mjs

@@ -1,4 +1,4 @@
-import { CHROME_UA, sleep } from './_seed-utils.mjs';
+import { CHROME_UA, sleep, resolveProxy, httpsProxyFetchRaw } from './_seed-utils.mjs';
 
 const MAX_RETRY_AFTER_MS = 60_000;
 const RETRYABLE_STATUSES = new Set([429, 503]);
@@ -41,6 +41,13 @@ export async function fetchOpenMeteoArchiveBatch(zones, opts) {
     maxRetries = 3,
     retryBaseMs = 2_000,
     label = zones.map((zone) => zone.name).join(', '),
+    // Test hooks. Production callers leave these unset; the helper uses the
+    // real proxy resolver + fetcher from _seed-utils.mjs. Tests inject mocks
+    // here to exercise the proxy fallback path without spinning up a real
+    // Decodo tunnel. Keep these undocumented in PR descriptions — they are
+    // implementation-only seams, not a public API surface.
+    _proxyResolver = resolveProxy,
+    _proxyFetcher = httpsProxyFetchRaw,
   } = opts;
 
   const params = new URLSearchParams({
@@ -53,6 +60,13 @@ export async function fetchOpenMeteoArchiveBatch(zones, opts) {
   });
   const url = `https://archive-api.open-meteo.com/v1/archive?${params.toString()}`;
 
+  // Track the last direct-path failure so the eventual throw carries useful
+  // context if proxy fallback is also unavailable / fails. Without this the
+  // helper would throw a generic "retries exhausted" message and lose the
+  // upstream error (timeout, ECONNRESET, HTTP status code) that triggered
+  // the fallback path.
+  let lastDirectError = null;
+
   for (let attempt = 0; attempt <= maxRetries; attempt++) {
     let resp;
     try {
@@ -61,13 +75,18 @@ export async function fetchOpenMeteoArchiveBatch(zones, opts) {
         signal: AbortSignal.timeout(timeoutMs),
       });
     } catch (err) {
+      lastDirectError = err;
       if (attempt < maxRetries) {
         const retryMs = retryBaseMs * 2 ** attempt;
         console.log(`  [OPEN_METEO] ${err?.message ?? err} for ${label}; retrying batch in ${Math.round(retryMs / 1000)}s`);
         await sleep(retryMs);
         continue;
       }
-      throw err;
+      // Final direct attempt threw (timeout, ECONNRESET, DNS, etc.). Fall
+      // through to the proxy fallback below — the previous version threw
+      // here, which silently bypassed the proxy path for thrown-error cases
+      // and only ran fallback for non-OK HTTP responses.
+      break;
     }
 
     if (resp.ok) {
@@ -78,15 +97,48 @@ export async function fetchOpenMeteoArchiveBatch(zones, opts) {
       return data;
     }
 
+    lastDirectError = new Error(`HTTP ${resp.status}`);
+
     if (RETRYABLE_STATUSES.has(resp.status) && attempt < maxRetries) {
       const retryMs = parseRetryAfterMs(resp.headers.get('retry-after')) ?? (retryBaseMs * 2 ** attempt);
       console.log(`  [OPEN_METEO] ${resp.status} for ${label}; retrying batch in ${Math.round(retryMs / 1000)}s`);
       await sleep(retryMs);
       continue;
     }
 
-    throw new Error(`Open-Meteo ${resp.status} for ${label}`);
+    // Direct attempt failed with non-retryable or after-final-retry status.
+    // Open-Meteo's free tier rate-limits per source IP; Railway containers
+    // share IP pools and hit 429 storms (logs.1776312819911 — every batch
+    // 429'd through 4 retries on 2026-04-16). Fall through to proxy fallback
+    // below before throwing.
+    break;
+  }
+
+  // Proxy fallback — same pattern as fredFetchJson / imfFetchJson in
+  // _seed-utils.mjs. Decodo gateway gets a different egress IP that is not
+  // (yet) on Open-Meteo's per-IP throttle. Skip silently if no proxy is
+  // configured (preserves existing behavior in non-Railway envs).
+  const proxyAuth = _proxyResolver();
+  if (proxyAuth) {
+    try {
+      console.log(`  [OPEN_METEO] direct exhausted on ${label} (${lastDirectError?.message ?? 'unknown'}); trying proxy`);
+      const { buffer } = await _proxyFetcher(url, proxyAuth, {
+        accept: 'application/json',
+        timeoutMs,
+      });
+      const data = normalizeArchiveBatchResponse(JSON.parse(buffer.toString('utf8')));
+      if (data.length !== zones.length) {
+        throw new Error(`Open-Meteo proxy batch size mismatch for ${label}: expected ${zones.length}, got ${data.length}`);
+      }
+      console.log(`  [OPEN_METEO] proxy succeeded for ${label}`);
+      return data;
+    } catch (proxyErr) {
+      console.warn(`  [OPEN_METEO] proxy fallback failed for ${label}: ${proxyErr?.message ?? proxyErr}`);
+    }
   }
 
-  throw new Error(`Open-Meteo retries exhausted for ${label}`);
+  throw new Error(
+    `Open-Meteo retries exhausted for ${label}${lastDirectError ? ` (last direct: ${lastDirectError.message})` : ''}`,
+    lastDirectError ? { cause: lastDirectError } : undefined,
+  );
 }

tests/open-meteo-proxy-fallback.test.mjs

@@ -0,0 +1,233 @@
+// Locks the proxy-fallback behavior added to _open-meteo-archive.mjs after
+// Railway 2026-04-16 logs showed seed-climate-zone-normals failing every
+// batch with HTTP 429 from Open-Meteo's per-IP free-tier throttle, with no
+// proxy retry.
+//
+// All HTTP is mocked — no real fetch / Decodo calls.
+
+import { test, beforeEach, afterEach } from 'node:test';
+import { strict as assert } from 'node:assert';
+
+process.env.UPSTASH_REDIS_REST_URL = 'https://redis.test';
+process.env.UPSTASH_REDIS_REST_TOKEN = 'fake-token';
+
+const ZONES = [
+  { name: 'Tropical', lat: 0,   lon: 0 },
+  { name: 'Polar',    lat: 80, lon: 0 },
+];
+
+const VALID_PAYLOAD = ZONES.map((z) => ({
+  latitude: z.lat,
+  longitude: z.lon,
+  daily: { time: ['2020-01-01'], temperature_2m_mean: [10] },
+}));
+
+const ARCHIVE_OPTS = {
+  startDate: '2020-01-01',
+  endDate: '2020-01-02',
+  daily: ['temperature_2m_mean'],
+  maxRetries: 1,
+  retryBaseMs: 10,
+  timeoutMs: 1000,
+};
+
+const originalFetch = globalThis.fetch;
+let capturedProxyCalls;
+
+beforeEach(() => {
+  capturedProxyCalls = [];
+});
+
+afterEach(() => {
+  globalThis.fetch = originalFetch;
+  delete process.env.PROXY_USER;
+  delete process.env.PROXY_PASS;
+  delete process.env.PROXY_HOST;
+  delete process.env.PROXY_PORT;
+  delete process.env.SEED_PROXY_AUTH;
+});
+
+// The helper accepts `_proxyResolver` and `_proxyFetcher` opt overrides
+// specifically for tests — production callers leave them unset and get the
+// real Decodo path from _seed-utils.mjs. This lets us exercise the proxy
+// branch without spinning up a real CONNECT tunnel.
+
+test('429 with no proxy configured: throws after exhausting retries (preserves pre-fix behavior)', async () => {
+  // Re-import per-test so module-level state (none currently) is fresh.
+  const { fetchOpenMeteoArchiveBatch } = await import(`../scripts/_open-meteo-archive.mjs?t=${Date.now()}`);
+
+  let calls = 0;
+  globalThis.fetch = async () => {
+    calls += 1;
+    return {
+      ok: false, status: 429,
+      headers: { get: () => null },
+      json: async () => ({}),
+    };
+  };
+
+  await assert.rejects(
+    () => fetchOpenMeteoArchiveBatch(ZONES, ARCHIVE_OPTS),
+    /Open-Meteo retries exhausted/,
+  );
+  // 1 initial + 1 retry (maxRetries=1) = 2 direct calls
+  assert.equal(calls, 2);
+});
+
+test('200 OK: returns parsed batch without touching proxy path', async () => {
+  const { fetchOpenMeteoArchiveBatch } = await import(`../scripts/_open-meteo-archive.mjs?t=${Date.now()}`);
+
+  globalThis.fetch = async () => ({
+    ok: true, status: 200,
+    headers: { get: () => null },
+    json: async () => VALID_PAYLOAD,
+  });
+
+  const result = await fetchOpenMeteoArchiveBatch(ZONES, ARCHIVE_OPTS);
+  assert.equal(result.length, 2);
+  assert.equal(result[0].latitude, 0);
+});
+
+test('batch size mismatch: throws even on 200', async () => {
+  const { fetchOpenMeteoArchiveBatch } = await import(`../scripts/_open-meteo-archive.mjs?t=${Date.now()}`);
+  globalThis.fetch = async () => ({
+    ok: true, status: 200,
+    headers: { get: () => null },
+    json: async () => [VALID_PAYLOAD[0]], // only 1, not 2
+  });
+  await assert.rejects(
+    () => fetchOpenMeteoArchiveBatch(ZONES, ARCHIVE_OPTS),
+    /batch size mismatch/,
+  );
+});
+
+test('non-retryable status (500): falls through to proxy attempt without extra retry', async () => {
+  const { fetchOpenMeteoArchiveBatch } = await import(`../scripts/_open-meteo-archive.mjs?t=${Date.now()}`);
+
+  let calls = 0;
+  globalThis.fetch = async () => {
+    calls += 1;
+    return {
+      ok: false, status: 500,
+      headers: { get: () => null },
+      json: async () => ({}),
+    };
+  };
+
+  await assert.rejects(
+    () => fetchOpenMeteoArchiveBatch(ZONES, ARCHIVE_OPTS),
+    /Open-Meteo retries exhausted/,
+  );
+  // Non-retryable status: no further retries — break out of the loop after
+  // first attempt, then the proxy-fallback block runs (no proxy env →
+  // skipped) → throws exhausted.
+  assert.equal(calls, 1);
+});
+
+// ─── Proxy fallback path — actually exercised via _proxyResolver/_proxyFetcher ───
+
+test('429 + proxy configured + proxy succeeds: returns proxy data, never throws', async () => {
+  const { fetchOpenMeteoArchiveBatch } = await import(`../scripts/_open-meteo-archive.mjs?t=${Date.now()}`);
+
+  globalThis.fetch = async () => ({
+    ok: false, status: 429,
+    headers: { get: () => null },
+    json: async () => ({}),
+  });
+
+  let proxyCalls = 0;
+  let receivedProxyAuth = null;
+  const result = await fetchOpenMeteoArchiveBatch(ZONES, {
+    ...ARCHIVE_OPTS,
+    _proxyResolver: () => 'user:pass@gate.decodo.com:7000',
+    _proxyFetcher: async (url, proxyAuth, _opts) => {
+      proxyCalls += 1;
+      receivedProxyAuth = proxyAuth;
+      assert.match(url, /archive-api\.open-meteo\.com\/v1\/archive\?/);
+      return { buffer: Buffer.from(JSON.stringify(VALID_PAYLOAD), 'utf8'), contentType: 'application/json' };
+    },
+  });
+
+  assert.equal(proxyCalls, 1);
+  assert.equal(receivedProxyAuth, 'user:pass@gate.decodo.com:7000');
+  assert.equal(result.length, 2);
+  assert.equal(result[1].latitude, 80);
+});
+
+test('thrown fetch error (timeout/ECONNRESET) on final direct attempt → proxy fallback runs (P1 fix)', async () => {
+  // Pre-fix bug: the catch block did `throw err` after the final direct retry,
+  // which silently bypassed proxy fallback for thrown-error cases (timeout,
+  // ECONNRESET, DNS). Lock the new control flow: thrown error → break →
+  // proxy fallback runs.
+  const { fetchOpenMeteoArchiveBatch } = await import(`../scripts/_open-meteo-archive.mjs?t=${Date.now()}`);
+
+  let directCalls = 0;
+  globalThis.fetch = async () => {
+    directCalls += 1;
+    throw Object.assign(new Error('Connect Timeout Error'), { code: 'UND_ERR_CONNECT_TIMEOUT' });
+  };
+
+  let proxyCalls = 0;
+  const result = await fetchOpenMeteoArchiveBatch(ZONES, {
+    ...ARCHIVE_OPTS,
+    _proxyResolver: () => 'user:pass@proxy.test:8000',
+    _proxyFetcher: async () => {
+      proxyCalls += 1;
+      return { buffer: Buffer.from(JSON.stringify(VALID_PAYLOAD), 'utf8'), contentType: 'application/json' };
+    },
+  });
+
+  assert.equal(directCalls, 2, 'direct attempts should exhaust retries before proxy');
+  assert.equal(proxyCalls, 1, 'proxy fallback MUST run on thrown-error path (regression guard)');
+  assert.equal(result.length, 2);
+});
+
+test('429 + proxy configured + proxy ALSO fails: throws exhausted with last direct error in cause', async () => {
+  const { fetchOpenMeteoArchiveBatch } = await import(`../scripts/_open-meteo-archive.mjs?t=${Date.now()}`);
+
+  globalThis.fetch = async () => ({
+    ok: false, status: 429,
+    headers: { get: () => null },
+    json: async () => ({}),
+  });
+
+  let proxyCalls = 0;
+  await assert.rejects(
+    () => fetchOpenMeteoArchiveBatch(ZONES, {
+      ...ARCHIVE_OPTS,
+      _proxyResolver: () => 'user:pass@proxy.test:8000',
+      _proxyFetcher: async () => {
+        proxyCalls += 1;
+        throw new Error('proxy 502');
+      },
+    }),
+    (err) => {
+      assert.match(err.message, /Open-Meteo retries exhausted/);
+      assert.match(err.message, /HTTP 429/);
+      return true;
+    },
+  );
+  assert.equal(proxyCalls, 1);
+});
+
+test('proxy fallback returns wrong batch size: caught + warns, throws exhausted', async () => {
+  const { fetchOpenMeteoArchiveBatch } = await import(`../scripts/_open-meteo-archive.mjs?t=${Date.now()}`);
+
+  globalThis.fetch = async () => ({
+    ok: false, status: 429,
+    headers: { get: () => null },
+    json: async () => ({}),
+  });
+
+  await assert.rejects(
+    () => fetchOpenMeteoArchiveBatch(ZONES, {
+      ...ARCHIVE_OPTS,
+      _proxyResolver: () => 'user:pass@proxy.test:8000',
+      _proxyFetcher: async () => ({
+        buffer: Buffer.from(JSON.stringify([VALID_PAYLOAD[0]]), 'utf8'),  // 1 instead of 2
+        contentType: 'application/json',
+      }),
+    }),
+    /Open-Meteo retries exhausted/,
+  );
+});