Conversation
Yahoo Finance throttles Railway egress IPs aggressively. 4 seeders (seed-commodity-quotes, seed-etf-flows, seed-gulf-quotes, seed-market-quotes) duplicated the same fetchYahooWithRetry block with no proxy fallback. This helper consolidates them and adds the proxy fallback. Yahoo-specific: CURL-ONLY proxy strategy. Probed 2026-04-16: query1.finance.yahoo.com via CONNECT (httpsProxyFetchRaw): HTTP 404 query1.finance.yahoo.com via curl (curlFetch): HTTP 200 Yahoo's edge blocks Decodo's CONNECT egress IPs but accepts the curl egress IPs. Helper deliberately omits the CONNECT leg — adding it would burn time on guaranteed-404 attempts. Production defaults expose ONLY curlProxyResolver + curlFetcher. All learnings from PR #3118 + #3119 reviews baked in: - lastDirectError accumulator across the loop, embedded in final throw + Error.cause chain - catch block uses break (NOT throw) so thrown errors also reach proxy - DI seams (_curlProxyResolver, _proxyCurlFetcher) for hermetic tests - _PROXY_DEFAULTS exported for production-default lock tests - Sync curlFetch wrapped with await Promise.resolve() to future-proof against an async refactor (Greptile P2 from #3119) Tests (tests/yahoo-fetch.test.mjs, 11 cases): - Production defaults: curl resolver/fetcher reference equality - Production defaults: NO CONNECT leg present (regression guard) - 200 OK passthrough, never touches proxy - 429 with no proxy → throws exhausted with HTTP 429 in message - Retry-After header parsed correctly - 429 + curl proxy succeeds → returns proxy data - Thrown fetch error on final retry → proxy fallback runs (P1 guard) - 429 + proxy ALSO fails → both errors visible in message + cause chain - Proxy malformed JSON → throws exhausted - Non-retryable 500 → no extra direct retry, falls to proxy - parseRetryAfterMs unit (exported sanity check) Verification: 11/11 helper tests pass. node --check clean. Phase 1 of 2 — seeder migrations follow.
Removes the duplicated fetchYahooWithRetry function (4 byte-identical
copies across seed-commodity-quotes, seed-etf-flows, seed-gulf-quotes,
seed-market-quotes) and routes all Yahoo Finance fetches through the
new scripts/_yahoo-fetch.mjs helper. Each seeder gains the curl-only
Decodo proxy fallback baked into the helper.
Per-seeder changes (mechanical):
- import { fetchYahooJson } from './_yahoo-fetch.mjs'
- delete the local fetchYahooWithRetry function
- replace 'const resp = await fetchYahooWithRetry(url, label); if (!resp)
return X; const json = await resp.json()' with
'let json; try { json = await fetchYahooJson(url, { label }); }
catch { return X; }'
- prune now-unused CHROME_UA/sleep imports where applicable
Latent bugs fixed in passing:
- seed-etf-flows.mjs:23 and seed-market-quotes.mjs:38 referenced
CHROME_UA without importing it (would throw ReferenceError at
runtime if the helper were called). Now the call site is gone in
etf-flows; in market-quotes CHROME_UA is properly imported because
Finnhub call still uses it.
seed-commodity-quotes also has fetchYahooChart1y (separate non-retry
function for gold history). Migrated to use fetchYahooJson under the
hood — preserves return shape, adds proxy fallback automatically.
Verification:
- node --check clean on all 4 modified seeders
- npm run typecheck:all clean
- npm run test:data: 5374/5374 pass
Phase 2 of 2.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…onest Retry-After test
Greptile P2: "[YAHOO] proxy (curl) succeeded" was logged BEFORE
JSON.parse(text). On malformed proxy JSON, Railway logs would show:
[YAHOO] proxy (curl) succeeded for AAPL
throw: Yahoo retries exhausted ...
Contradictory + breaks the post-deploy log-grep verification this PR
relies on ("look for [YAHOO] proxy (curl) succeeded"). Fix: parse
first; success log only fires when parse succeeds AND the value is
about to be returned.
Greptile P3: 'Retry-After header parsed correctly' test used header
value '0', but parseRetryAfterMs() treats non-positive seconds as null
→ helper falls through to default linear backoff. So the test was
exercising the wrong branch despite its name.
Fix: added _sleep DI opt seam to the helper. New test injects a sleep
spy and asserts the captured duration:
Retry-After: '7' → captured sleep == [7000] (Retry-After branch)
no Retry-After → captured sleep == [10] (default backoff = retryBaseMs * 1)
Two paired tests lock both branches separately so a future regression
that collapses them is caught.
Also added a log-ordering regression test: malformed proxy JSON must
NOT emit the 'succeeded' log. Captures console.log into an array and
asserts no 'proxy (curl) succeeded' line appeared before the throw.
Verification:
- tests/yahoo-fetch.test.mjs: 13/13 (was 11, +2)
- npm run test:data: 5376/5376 (+2)
- npm run typecheck:all: clean
Followup commits on PR #3120.
Owner
Author
|
Both findings addressed in 1edc2a2. P2 — log ordering: success log moved to AFTER P3 — Retry-After test now actually exercises the branch. Added
Paired tests lock both branches separately so a future regression collapsing them gets caught. Tests: 13/13 (was 11, +2). test:data 5376/5376 (+2). typecheck:all clean. Learnings to forward to PR C (GDELT)
|
Contributor
Greptile SummaryThis PR consolidates four byte-identical inline Confidence Score: 5/5Safe to merge — no P0/P1 findings; all remaining observations are cosmetic or pre-existing. The helper's control flow (break-not-throw, lastDirectError accumulator, DI seams, Error.cause chaining) is correct. All four seeder migrations correctly preserve null-return semantics via try/catch wrappers. Latent CHROME_UA ReferenceError bugs are fixed. The 11-case test suite locks every design invariant from previous review cycles, including the production-defaults reference-equality check and the P1 regression guard for the throw-bypass path. No correctness, data-integrity, or security issues were found. No files require special attention. Important Files Changed
Sequence DiagramsequenceDiagram
participant S as Seeder
participant H as fetchYahooJson
participant Y as Yahoo Finance
participant R as resolveProxy
participant C as curlFetch
S->>H: fetchYahooJson(url, opts)
loop attempt 0..maxRetries
H->>Y: fetch(url) with UA + timeout
alt 200 OK
Y-->>H: resp.ok = true
H-->>S: return resp.json()
else 429 or 503 and not last attempt
Y-->>H: retryable status
H->>H: sleep(retryAfter or linear backoff)
else non-retryable or last attempt
Y-->>H: non-retryable status
H->>H: set lastDirectError, break
else fetch throws
Y-->>H: throw timeout or network error
H->>H: set lastDirectError, retry or break
end
end
H->>R: _curlProxyResolver()
alt no proxy configured
R-->>H: null
H-->>S: throw retries exhausted with last direct error
else proxy available
R-->>H: proxy auth string
H->>C: curlFetch(url, proxyAuth, headers)
alt curl succeeds
C-->>H: JSON text
H-->>S: return JSON.parse(text)
else curl fails
C-->>H: throw curlErr
H-->>S: throw exhausted with direct and proxy errors in cause chain
end
end
Reviews (1): Last reviewed commit: "fix(_yahoo-fetch): log success AFTER par..." | Re-trigger Greptile |
koala73
added a commit
that referenced
this pull request
Apr 16, 2026
…d API GDELT (api.gdeltproject.org) is a public free API with strict per-IP throttling. seed-gdelt-intel currently has no proxy fallback — Railway egress IPs hit 429 storms and the seeder degrades. Probed 2026-04-16: Decodo curl egress against GDELT gives ~40% success per attempt (session-rotates IPs per call). Helper retries up to 5×; expected overall success ~92% (1 - 0.6^5). PROXY STRATEGY — CURL ONLY WITH MULTI-RETRY Differs from _yahoo-fetch.mjs (single proxy attempt) and _open-meteo-archive.mjs (CONNECT + curl cascade): - Curl-only: CONNECT not yet probed cleanly against GDELT. - Multi-retry on the curl leg: the proxy IS the rotation mechanism (each call → different egress IP), so successive attempts probe different IPs in the throttle pool. - Distinguishes retryable (HTTP 429/503 from upstream) from non-retryable (parse failure, auth, network) — bails immediately on non-retryable to avoid 5× of wasted log noise. Direct loop uses LONGER backoff than Yahoo's 5s base (10s) — GDELT's throttle window is wider than Yahoo's, so quick retries usually re-hit the same throttle. Tests (tests/gdelt-fetch.test.mjs, 13 cases — every learning from PR #3118 + #3119 + #3120 baked in): - Production defaults: curl resolver/fetcher reference equality - Production defaults: NO CONNECT leg (regression guard for unverified path) - 200 OK passthrough - 429 with no proxy → throws with HTTP 429 in message - Retry-After parsed (DI _sleep capture asserts 7000ms not retryBaseMs) - Retry-After absent → linear backoff retryBaseMs (paired branch test) - **Proxy multi-retry: 4× HTTP 429 then 5th succeeds → returns data** (asserts 5 proxy calls + 4 inter-proxy backoffs of proxyRetryBaseMs) - **Proxy non-retryable (parse failure) bails after 1 attempt** (does NOT burn all proxyMaxAttempts on a structural failure) - **Proxy retryable + non-retryable mix: retries on 429, bails on parse** - Thrown fetch error on final retry → proxy multi-retry runs (P1 guard) - All proxy attempts fail → throws with 'X/N attempts' in message + cause - Malformed JSON does NOT emit succeeded log before throw (P2 guard) - parseRetryAfterMs unit Verification: - tests/gdelt-fetch.test.mjs → 13/13 pass - node --check scripts/_gdelt-fetch.mjs → clean Phase 1 of 2. Seeder migration follows.
Merged
5 tasks
koala73
added a commit
that referenced
this pull request
Apr 16, 2026
…delt-intel migration (#3122) * feat(_gdelt-fetch): curl proxy multi-retry helper for per-IP-throttled API GDELT (api.gdeltproject.org) is a public free API with strict per-IP throttling. seed-gdelt-intel currently has no proxy fallback — Railway egress IPs hit 429 storms and the seeder degrades. Probed 2026-04-16: Decodo curl egress against GDELT gives ~40% success per attempt (session-rotates IPs per call). Helper retries up to 5×; expected overall success ~92% (1 - 0.6^5). PROXY STRATEGY — CURL ONLY WITH MULTI-RETRY Differs from _yahoo-fetch.mjs (single proxy attempt) and _open-meteo-archive.mjs (CONNECT + curl cascade): - Curl-only: CONNECT not yet probed cleanly against GDELT. - Multi-retry on the curl leg: the proxy IS the rotation mechanism (each call → different egress IP), so successive attempts probe different IPs in the throttle pool. - Distinguishes retryable (HTTP 429/503 from upstream) from non-retryable (parse failure, auth, network) — bails immediately on non-retryable to avoid 5× of wasted log noise. Direct loop uses LONGER backoff than Yahoo's 5s base (10s) — GDELT's throttle window is wider than Yahoo's, so quick retries usually re-hit the same throttle. Tests (tests/gdelt-fetch.test.mjs, 13 cases — every learning from PR #3118 + #3119 + #3120 baked in): - Production defaults: curl resolver/fetcher reference equality - Production defaults: NO CONNECT leg (regression guard for unverified path) - 200 OK passthrough - 429 with no proxy → throws with HTTP 429 in message - Retry-After parsed (DI _sleep capture asserts 7000ms not retryBaseMs) - Retry-After absent → linear backoff retryBaseMs (paired branch test) - **Proxy multi-retry: 4× HTTP 429 then 5th succeeds → returns data** (asserts 5 proxy calls + 4 inter-proxy backoffs of proxyRetryBaseMs) - **Proxy non-retryable (parse failure) bails after 1 attempt** (does NOT burn all proxyMaxAttempts on a structural failure) - **Proxy retryable + non-retryable mix: retries on 429, bails on parse** - Thrown fetch error on final retry → proxy multi-retry runs (P1 guard) - All proxy attempts fail → throws with 'X/N attempts' in message + cause - Malformed JSON does NOT emit succeeded log before throw (P2 guard) - parseRetryAfterMs unit Verification: - tests/gdelt-fetch.test.mjs → 13/13 pass - node --check scripts/_gdelt-fetch.mjs → clean Phase 1 of 2. Seeder migration follows. * feat(seed-gdelt-intel): migrate to _gdelt-fetch helper Replaces direct fetch + ad-hoc retry in seed-gdelt-intel with the new fetchGdeltJson helper. Each topic call now gets: 3 direct retries (10/20/40s backoff) → 5 curl proxy attempts via Decodo session-rotating egress. Specific changes: - import fetchGdeltJson from _gdelt-fetch.mjs - fetchTopicArticles: replace fetch+retry+throw block with single await fetchGdeltJson(url, { label: topic.id }) - fetchTopicTimeline: same — best-effort try/catch returns [] on any failure (preserved). Helper still attempts proxy fallback before throwing, so a 429-throttled IP doesn't kill the timeline. - fetchWithRetry: collapsed from outer 3-retry loop with 60/120/240s backoff (which would have multiplied to 24 attempts/topic on top of helper's 8) to a thin wrapper that translates exhaustion into the {exhausted, articles:[]} shape the caller uses to drive POST_EXHAUST_DELAY_MS cooldown. - Drop CHROME_UA import (no longer used directly; helper handles it). Helper's exhausted-throw includes 'HTTP 429' substring when 429 was the upstream signal, so the existing is429 detection in fetchWithRetry continues to work without modification. Verification: - node --check scripts/seed-gdelt-intel.mjs → clean - npm run typecheck:all → clean - npm run test:data → 5382/5382 (was 5363, +13 from helper + 6 from prior PR work) Phase 2 of 2. * fix(_gdelt-fetch): proxy timeouts/network errors RETRY (rotates Decodo session) P1 from PR #3122 review: probed Decodo curl egress against GDELT (2026-04-16) gave 200/200/429/TIMEOUT/429 — TIMEOUT is part of the normal transient mix that the multi-retry design exists to absorb. Pre-fix logic only retried on substring 'HTTP 429'/'HTTP 503' matches, so a curl exec timeout (Node Error with no .status, not a SyntaxError) bailed on the first attempt. The PR's headline 'expected ~92% success with 5 attempts' was therefore not actually achievable for one of the exact failure modes that motivated the design. Reframed the proxy retryability decision around what we CAN reliably discriminate from the curl error shape: curlErr.status == number → retry only if 429/503 (curlFetch attaches .status only when curl returned a clean HTTP status) curlErr instanceof SyntaxError → bail (parse failure is structural) otherwise → RETRY (timeout, ECONNRESET, DNS, curl exec failure, CONNECT tunnel failure — all transient; rotating Decodo session usually clears them) P2 from same review: tests covered HTTP-status proxy retries + parse failures but never the timeout/thrown-error class. Added 3 tests: - proxy timeout (no .status) RETRIES → asserts proxyCalls=2 after a first-attempt ETIMEDOUT then second-attempt success - proxy ECONNRESET (no .status) RETRIES → same pattern - proxy HTTP 4xx with .status (e.g. 401 auth) does NOT retry → bails after 1 attempt Existing tests still pass — they use 'HTTP 429' Error WITHOUT .status, which now flows through the 'else: assume transient' branch and still retries. Only differences: the regex parsing is gone and curlFetch's .status property is the canonical signal. Verification: - tests/gdelt-fetch.test.mjs: 16/16 (was 13, +3) - npm run test:data: 5385/5385 (+3) - npm run typecheck:all: clean Followup commit on PR #3122. * fix(seed-gdelt-intel): timeline calls fast-fail (maxRetries:0, proxyMaxAttempts:0) P1 from PR #3122 review: fetchTopicTimeline is best-effort (returns [] on any failure), but the migration routed it through fetchGdeltJson with the helper's article-fetch defaults: 3 direct retries (10/20/40s backoff = ~70s) + 5 proxy attempts (5s base = ~20s) = ~90s worst case per call. Called 2× per topic × 6 topics = 12 calls = up to ~18 minutes of blocking on data the seeder discards on failure. Pre-helper code did a single direct fetch with no retry. Real operational regression under exactly the GDELT 429 storm conditions this PR is meant to absorb. Fix: 1. seed-gdelt-intel.mjs:fetchTopicTimeline now passes maxRetries:0, proxyMaxAttempts:0 — single direct attempt, no proxy, throws on first failure → caught, returns []. Matches pre-helper timing exactly. Article fetches keep the full retry budget; only timelines fast-fail. 2. _gdelt-fetch.mjs gate: skip the proxy block entirely when proxyMaxAttempts <= 0. Pre-fix the 'trying proxy (curl) up to 0×' log line would still emit even though the for loop runs zero times, producing a misleading line that the proxy was attempted when it wasn't. Tests (2 new): - maxRetries:0 + proxyMaxAttempts:0 → asserts directCalls=1, proxyCalls=0 even though _curlProxyResolver returns a valid auth string (proxy block must be fully bypassed). - proxyMaxAttempts:0 → captures console.log and asserts no 'trying proxy' line emitted (no misleading 'up to 0×' line). Verification: - tests/gdelt-fetch.test.mjs: 18/18 (was 16, +2) - npm run test:data: 5387/5387 (+2) - npm run typecheck:all: clean Followup commit on PR #3122. * fix(gdelt): direct parse-failure reaches proxy + timeline budget tweak + JSDoc accuracy 3 Greptile P2s on PR #3122: P2a — _gdelt-fetch.mjs:112: `resp.json()` was called outside the try/catch that guards fetch(). A 200 OK with HTML/garbage body (WAF challenge, partial response, gzip mismatch) would throw SyntaxError and escape the helper entirely — proxy fallback never ran. The proxy leg already parsed inside its own catch; making the direct leg symmetric. New regression test: direct 200 OK with malformed JSON must reach the proxy and recover. P2b — seed-gdelt-intel.mjs timeline budget bumped from 0/0 to 0/2. Best-effort timelines still fast-fail on direct 429 (no direct retries) but get 2 proxy attempts via Decodo session rotation before returning []. Worst case: ~25s/call × 12 calls = ~5 min ceiling under heavy throttling vs ~3 min with 0/0. Tradeoff: small additional time budget for a real chance to recover timeline data via proxy IP rotation. Articles still keep the full retry budget. P2c — JSDoc said 'Linear proxy backoff base' but the implementation uses a flat constant (proxyRetryBaseMs, line 156). Linear growth would not help here because Decodo rotates the session IP per call — the next attempt's success is independent of the previous wait. Doc now reads 'Fixed (constant, NOT linear) backoff' with the rationale. Verification: - tests/gdelt-fetch.test.mjs: 19/19 pass (was 18, +1) - npm run test:data: 5388/5388 (+1) - npm run typecheck:all: clean Followup commit on PR #3122. * test(gdelt): clarify helper-API vs seeder-mirror tests + add 0/2 lock Reviewer feedback on PR #3122 conflated two test classes: - Helper-API tests (lock the helper's contract for arbitrary callers using budget knobs like 0/0 — independent of any specific seeder) - Seeder-mirror tests (lock the budget the actual production caller in seed-gdelt-intel.mjs uses) Pre-fix the test file only had the 0/0 helper-API tests, with a section header that read 'Best-effort caller budgets (fast-fail)' — ambiguous about whether 0/0 was the helper API contract or the seeder's choice. Reviewer assumed seeder still used 0/0 because the tests locked it, but seed-gdelt-intel.mjs:97-98 actually uses 0/2 (per the prior P2b fix). Fixes: 1. Section header for the 0/0 tests now explicitly says these are helper-API tests and notes that seed-gdelt-intel uses 0/2 (not 0/0). Eliminates the conflation. 2. New 'Seeder-mirror: 0/2' section with 2 tests that lock the seeder's actual choice end-to-end: - 0/2 with first proxy attempt 429 + second succeeds → returns data (asserts directCalls=1, proxyCalls=2) - 0/2 with both proxy attempts failing → throws exhausted with '2/2 attempts' in message (asserts the budget propagates to the error message correctly) These tests would catch any future regression where the seeder's 0/2 choice gets reverted to 0/0 OR where the helper stops honoring the proxyMaxAttempts override. Verification: - tests/gdelt-fetch.test.mjs: 21/21 (was 19, +2) - npm run test:data: 5390/5390 (+2) - npm run typecheck:all: clean Followup commit on PR #3122.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
4 seeders (
seed-commodity-quotes,seed-etf-flows,seed-gulf-quotes,seed-market-quotes) all fetch from Yahoo Finance with byte-identical localfetchYahooWithRetryhelpers and no proxy fallback. Yahoo throttles Railway egress IPs aggressively (429s); current behavior degrades tonullafter 4 direct retries.What
New
scripts/_yahoo-fetch.mjshelper. 4 seeders migrated to use it.Yahoo-specific design: curl-ONLY proxy strategy
Probed 2026-04-16:
httpsProxyFetchRawviagate.decodo.com)curlFetchviaus.decodo.com)Yahoo's edge blocks Decodo's CONNECT egress IP pool. Helper deliberately omits the CONNECT leg — adding it would burn time on guaranteed-404 attempts. Production defaults expose ONLY
curlProxyResolver+curlFetcher.If Yahoo's behavior toward Decodo CONNECT ever changes, follow
scripts/_open-meteo-archive.mjscascade pattern to add a CONNECT leg.Learnings baked in (all from PR #3118 + #3119 reviews)
lastDirectErroraccumulator across the loop, embedded in final throw +Error.causechainbreak(NOTthrow) so thrown errors also reach the proxy path_curlProxyResolver,_proxyCurlFetcher) for hermetic tests_PROXY_DEFAULTSexported for production-default lock tests (catches wiring regressions)curlFetchwrapped withawait Promise.resolve()— no-op today, future-proof against async refactorLatent bugs fixed in passing
seed-etf-flows.mjs:23andseed-market-quotes.mjs:38referencedCHROME_UAwithout importing it (wouldReferenceErrorif the helper were ever called). Migration removes the call site in etf-flows; properly imports CHROME_UA in market-quotes (still used by Finnhub fetch).Test plan
tests/yahoo-fetch.test.mjs— 11 cases:npm run test:data→ 5374/5374 (was 5358, +16: 11 yahoo + 5 from prior infra)npm run typecheck:all→ cleannode --checkclean on all 4 modified seedersseed-bundle-market-backup(covers all 4) → look for[YAHOO] proxy (curl) succeededlog lines on Railway IPs that hit 429 directlyFollowup
PR C (GDELT —
scripts/_gdelt-fetch.mjs) follows next, applying the same pattern but with aggressive proxy retries since Decodo's session-rotating egress hits GDELT's per-IP throttle ~60% of the time.