Description
isTrackingParam in src/search/core/recent-cache-dedup.ts:29 compares the raw query-param name case-sensitively (TRACKING_PARAM_EXACT.has(name), name.startsWith('utm_')), but the file header (src/search/core/recent-cache-dedup.ts:3-6) documents that normalization "removes tracking params (utm_*, gclid, fbclid)". Uppercase variants (?UTM_SOURCE=x, ?GCLID=x) therefore survive normalization and produce different dedup keys for the same page, so dedupAgainstRecentUrls (src/search/core/recent-cache-dedup.ts:97) misses the duplicate. Host handling in the same function is lowercased, so the param check is inconsistent.
Reproduce
- Run from the repo root:
npx --yes tsx -e "import {normalizeUrlForDedup,dedupAgainstRecentUrls} from './src/search/core/recent-cache-dedup.ts'; const a=normalizeUrlForDedup('https://x.com/p?utm_source=news'); const b=normalizeUrlForDedup('https://x.com/p?UTM_SOURCE=news'); const c=normalizeUrlForDedup('https://x.com/p?GCLID=abc'); const d=normalizeUrlForDedup('https://x.com/p?gclid=abc'); console.log(JSON.stringify({a,b,c,d})); const kept=dedupAgainstRecentUrls([{url:'https://x.com/p?UTM_SOURCE=news',title:'t',snippet:'s',engine:'e',relevance_score:1}],['https://x.com/p']); console.log('kept='+kept.length);"
- Observed:
{"a":"https://x.com/p","b":"https://x.com/p?UTM_SOURCE=news","c":"https://x.com/p?GCLID=abc","d":"https://x.com/p"} and kept=1 — lowercase tracking params are stripped but uppercase ones are kept, and the duplicate is not filtered (should be kept=0).
Expected
normalizeUrlForDedup('https://x.com/p?UTM_SOURCE=news') should return https://x.com/p and the dedupAgainstRecentUrls call above should return length 0. Smallest fix: lowercase the param name before the tracking check in isTrackingParam (e.g. const n = name.toLowerCase()), keeping original casing for retained params.
Checklist
Description
isTrackingParaminsrc/search/core/recent-cache-dedup.ts:29compares the raw query-param name case-sensitively (TRACKING_PARAM_EXACT.has(name),name.startsWith('utm_')), but the file header (src/search/core/recent-cache-dedup.ts:3-6) documents that normalization "removes tracking params (utm_*, gclid, fbclid)". Uppercase variants (?UTM_SOURCE=x,?GCLID=x) therefore survive normalization and produce different dedup keys for the same page, sodedupAgainstRecentUrls(src/search/core/recent-cache-dedup.ts:97) misses the duplicate. Host handling in the same function is lowercased, so the param check is inconsistent.Reproduce
npx --yes tsx -e "import {normalizeUrlForDedup,dedupAgainstRecentUrls} from './src/search/core/recent-cache-dedup.ts'; const a=normalizeUrlForDedup('https://x.com/p?utm_source=news'); const b=normalizeUrlForDedup('https://x.com/p?UTM_SOURCE=news'); const c=normalizeUrlForDedup('https://x.com/p?GCLID=abc'); const d=normalizeUrlForDedup('https://x.com/p?gclid=abc'); console.log(JSON.stringify({a,b,c,d})); const kept=dedupAgainstRecentUrls([{url:'https://x.com/p?UTM_SOURCE=news',title:'t',snippet:'s',engine:'e',relevance_score:1}],['https://x.com/p']); console.log('kept='+kept.length);"{"a":"https://x.com/p","b":"https://x.com/p?UTM_SOURCE=news","c":"https://x.com/p?GCLID=abc","d":"https://x.com/p"}andkept=1— lowercase tracking params are stripped but uppercase ones are kept, and the duplicate is not filtered (should bekept=0).Expected
normalizeUrlForDedup('https://x.com/p?UTM_SOURCE=news')should returnhttps://x.com/pand thededupAgainstRecentUrlscall above should return length0. Smallest fix: lowercase the param name before the tracking check inisTrackingParam(e.g.const n = name.toLowerCase()), keeping original casing for retained params.Checklist