fix: refuse an already-rejected token without a second argon2 verification

docs/overview.md

@@ -130,6 +130,7 @@ CSRF protection uses a token the server owns for its lifetime:
 4. **Client Hashing**: Client hashes token before sending in requests (never sends raw token)
 5. **Verification**: Server compares request hash against the token it is holding
 6. **Caching**: Verified tokens stored in `server/verifiedTokens.ts` (in-memory `Map` of token to last-seen time) to avoid redundant cryptographic operations
+7. **Rejection Caching**: Tokens that fail a completed verification are kept in a bounded set (`server/rejectedTokens.ts`) until it evicts them at the cap, so a replay of a dead token is refused without a second argon2 verification; a verification that never produced a result, because the hash could not be parsed or the token file could not be read, leaves nothing behind
 
 Rewriting the token file under a running server does not re-key it: the server keeps the token it is already handing out, and logs once that the file diverged if a request is rejected while it has.
 
@@ -213,6 +214,7 @@ Key server-side modules:
 - **`server/pageContentService.ts`**: Reads result pages for answer grounding: SSRF-guarded fetches with a byte cap, readable-text extraction, and query-relevant passage selection.
 - **`server/searchToken.ts`**: Manages a token at `{os.tempdir()}/minisearch-token` used for CSRF protection on search requests.
 - **`server/verifiedTokens.ts`**: In-memory `Map` of verified session token to last-seen time, evicted after 30 idle minutes, plus a cumulative count of the distinct sessions seen since the last restart.
+- **`server/rejectedTokens.ts`**: Bounded in-memory set of tokens that already failed a completed verification, so a replay is refused without a second argon2 check until the set evicts it at the cap; a token refused once cannot become valid in the same process, so the set is exact, and a token that never got a verification result never occupies a slot.
 - **`server/searchesSinceLastRestart.ts`**: In-memory counters for search analytics.
 
 ### Cache Control
@@ -310,10 +312,13 @@ verification, the funnel `/search/text`, `/search/images`, `/page-content` and
 | `reasons.missingToken` | number | No token on the request, which is what an outdated client looks like |
 | `reasons.invalidToken` | number | A token that failed verification, which is what probing looks like |
 | `bySurface` | object | `authorized` and `rejected` per endpoint family: `search`, `pageContent`, `inference`, `other` |
+| `rejectedTokenCacheHits` | number | Rejections served from the rejected-token cache without a second argon2 verification |
 | `limiter` | object | The limiter's `points` and `durationSeconds`, without which a rejection count says nothing |
 
 `authorized` plus every entry of `reasons` sums to `requests`, and each half of
 `bySurface` sums to its side of that, on the same principle as `pageReads`.
+`rejectedTokenCacheHits` counts a subset of `reasons.invalidToken` rather
+than adding to that sum.
 
 The limiter keys on the client IP and none of that reaches these counters: no
 address, no token, no query, no per-request timestamp. The cut is by reason and

docs/security.md

@@ -38,6 +38,7 @@ Every HTTP request from client to backend carries a `token` query parameter for
 3. **Per-Request Auth**: Client includes token as `?token=` parameter on all `/search/text`, `/search/images` and `/page-content` requests
 4. **Server Verification**: `handleTokenVerification()` in `searchEndpointServerHook.ts` validates the token before proxying to SearXNG
 5. **Session Tracking**: Validated tokens are stored in an in-memory `Set<string>` (`verifiedTokens.ts`) for session counting
+6. **Rejection Caching**: Tokens that fail a completed verification are kept in a bounded in-memory set (`rejectedTokens.ts`) until it evicts them at the cap, so a replay is refused without paying for a second argon2 verification; a token whose verification threw instead of returning a result, whether from an unparseable hash or an unreadable token file, is refused without taking a slot; how many rejections were served that way is reported on `/status` (`authorization.rejectedTokenCacheHits`)
 
 ## Privacy
 
@@ -74,6 +75,7 @@ Every HTTP request from client to backend carries a `token` query parameter for
 |--------|---------|
 | `server/searchToken.ts` | Reads/writes the CSRF token from `{tempdir}/minisearch-token` |
 | `server/verifiedTokens.ts` | In-memory `Set<string>` of verified session tokens |
+| `server/rejectedTokens.ts` | Bounded in-memory set of tokens that already failed a completed verification, so a replay skips the second argon2 check |
 | `server/searchesSinceLastRestart.ts` | In-memory counters for aggregate search outcomes (text/image search totals, and how often searches came back empty or were fully discarded), reported on `/status` |
 | `server/pageReadsSinceLastRestart.ts` | In-memory aggregate counters for pages read for grounding (outcomes, durations, passage ratios), reported on `/status`; records no query, URL, host, or per-read timestamp |
 | `server/searchEndpointServerHook.ts` | Proxies text/image search to SearXNG after token verification (via `handleTokenVerification`) |

server/authorizationSinceLastRestart.ts

@@ -8,6 +8,7 @@
  * share it, none of which needs to know who asked.
  */
 
+import { getRejectedTokenCacheHits } from "./rejectedTokens.ts";
 import {
   RATE_LIMIT_DURATION_SECONDS,
   RATE_LIMIT_POINTS,
@@ -78,6 +79,7 @@ export function getAuthorizationStats() {
     requests,
     authorized,
     rejectedRate: Number(((rejected / requests) * 100 || 0).toFixed(1)),
+    rejectedTokenCacheHits: getRejectedTokenCacheHits(),
     reasons: { ...reasons },
     // Deep copy, so a caller holding a snapshot for comparison does not watch
     // it change under them as later requests arrive.

server/handleTokenVerification.test.ts

@@ -177,6 +177,7 @@ describe("handleTokenVerification", () => {
       "limiter",
       "reasons",
       "rejectedRate",
+      "rejectedTokenCacheHits",
       "requests",
     ]);
     expect(Object.keys(stats.reasons).sort()).toEqual([

server/rejectedTokens.test.ts

@@ -0,0 +1,43 @@
+import { beforeEach, describe, expect, it, vi } from "vitest";
+
+describe("rejectedTokens", () => {
+  beforeEach(() => {
+    vi.resetModules();
+  });
+
+  it("refuses a token it recorded and nothing else", async () => {
+    const { addRejectedToken, isRejectedToken } = await import(
+      "./rejectedTokens"
+    );
+
+    expect(isRejectedToken("a")).toBe(false);
+    addRejectedToken("a");
+    expect(isRejectedToken("a")).toBe(true);
+    expect(isRejectedToken("b")).toBe(false);
+  });
+
+  it("evicts the oldest token once the cap is reached, so the set stays bounded", async () => {
+    const { addRejectedToken, isRejectedToken, MAX_REJECTED_TOKENS } =
+      await import("./rejectedTokens");
+
+    for (let i = 0; i <= MAX_REJECTED_TOKENS; i++) {
+      addRejectedToken(`token-${i}`);
+    }
+
+    expect(isRejectedToken("token-0")).toBe(false);
+    expect(isRejectedToken(`token-${MAX_REJECTED_TOKENS}`)).toBe(true);
+  });
+
+  it("does not evict anything when an already recorded token is recorded again at the cap", async () => {
+    const { addRejectedToken, isRejectedToken, MAX_REJECTED_TOKENS } =
+      await import("./rejectedTokens");
+
+    for (let i = 0; i < MAX_REJECTED_TOKENS; i++) {
+      addRejectedToken(`token-${i}`);
+    }
+    addRejectedToken("token-5");
+
+    expect(isRejectedToken("token-0")).toBe(true);
+    expect(isRejectedToken(`token-${MAX_REJECTED_TOKENS - 1}`)).toBe(true);
+  });
+});

server/rejectedTokens.ts

@@ -0,0 +1,45 @@
+/**
+ * Tokens that have already failed argon2 verification.
+ *
+ * A process verifies against one search token for its whole life, so a token
+ * that failed once cannot become valid again in this process: the lookup is
+ * exact rather than a heuristic. A replay of a dead token then costs a Set
+ * check instead of a full argon2id verification, which is what a stream of
+ * dead search URLs from earlier deployments used to pay.
+ */
+const rejectedTokens = new Set<string>();
+
+/**
+ * Bounds the set against a flood of distinct junk tokens: once full, the
+ * oldest entry is evicted and that token pays for one verification again if
+ * it comes back.
+ */
+export const MAX_REJECTED_TOKENS = 1024;
+
+/**
+ * Rejections served from this set without a second argon2 verification, so
+ * `/status` can show how much of the dead-token traffic stopped costing a
+ * verification.
+ */
+let rejectedTokenCacheHits = 0;
+
+export function isRejectedToken(token: string) {
+  return rejectedTokens.has(token);
+}
+
+export function addRejectedToken(token: string) {
+  if (rejectedTokens.has(token)) return;
+  if (rejectedTokens.size >= MAX_REJECTED_TOKENS) {
+    const oldest = rejectedTokens.values().next().value;
+    if (oldest !== undefined) rejectedTokens.delete(oldest);
+  }
+  rejectedTokens.add(token);
+}
+
+export function recordRejectedTokenCacheHit(): void {
+  rejectedTokenCacheHits++;
+}
+
+export function getRejectedTokenCacheHits(): number {
+  return rejectedTokenCacheHits;
+}

server/verifyTokenAndRateLimit.test.ts

@@ -79,6 +79,105 @@ describe("verifyTokenAndRateLimit", () => {
     });
   });
 
+  it("refuses an already rejected token without a second argon2 verification", async () => {
+    mockArgon2VerifyResult = false;
+    vi.resetModules();
+    const { verifyTokenAndRateLimit } = await import(
+      "./verifyTokenAndRateLimit"
+    );
+    const hashWasm = await import("hash-wasm");
+    const { getAuthorizationStats } = await import(
+      "./authorizationSinceLastRestart"
+    );
+
+    const invalid = {
+      isAuthorized: false,
+      statusCode: 401,
+      error: "Invalid token.",
+      reason: "invalidToken",
+    };
+    const cacheHitsBefore = getAuthorizationStats().rejectedTokenCacheHits;
+
+    expect(await verifyTokenAndRateLimit("dead-token")).toEqual(invalid);
+    expect(await verifyTokenAndRateLimit("dead-token")).toEqual(invalid);
+
+    // The first refusal pays for the verification; the repeat is a cache hit.
+    expect(hashWasm.argon2Verify).toHaveBeenCalledTimes(1);
+    expect(getAuthorizationStats().rejectedTokenCacheHits).toBe(
+      cacheHitsBefore + 1,
+    );
+  });
+
+  it("does not record a token that verifies", async () => {
+    mockArgon2VerifyResult = true;
+    vi.resetModules();
+    const { verifyTokenAndRateLimit } = await import(
+      "./verifyTokenAndRateLimit"
+    );
+    const { isRejectedToken } = await import("./rejectedTokens");
+
+    const result = await verifyTokenAndRateLimit("live-token");
+    expect(result.isAuthorized).toBe(true);
+    expect(isRejectedToken("live-token")).toBe(false);
+  });
+
+  it("does not cache a token whose verification threw, so the same token verifies on the next attempt", async () => {
+    mockArgon2VerifyResult = true;
+    vi.resetModules();
+    const { verifyTokenAndRateLimit } = await import(
+      "./verifyTokenAndRateLimit"
+    );
+    const hashWasm = await import("hash-wasm");
+    const searchToken = await import("./searchToken");
+    const { isRejectedToken } = await import("./rejectedTokens");
+    // The transient throw: the token file could not be read, not the token
+    // being dead.
+    vi.mocked(searchToken.getSearchToken).mockImplementationOnce(() => {
+      throw new Error("ENOENT");
+    });
+
+    const invalid = {
+      isAuthorized: false,
+      statusCode: 401,
+      error: "Invalid token.",
+      reason: "invalidToken",
+    };
+    expect(await verifyTokenAndRateLimit("flaky-token")).toEqual(invalid);
+
+    // Nothing was cached, so once the read succeeds the same token verifies.
+    expect((await verifyTokenAndRateLimit("flaky-token")).isAuthorized).toBe(
+      true,
+    );
+    expect(hashWasm.argon2Verify).toHaveBeenCalledTimes(1);
+    expect(isRejectedToken("flaky-token")).toBe(false);
+  });
+
+  it("does not cache a token when argon2 itself throws, as it does on an unparseable hash", async () => {
+    vi.resetModules();
+    const { verifyTokenAndRateLimit } = await import(
+      "./verifyTokenAndRateLimit"
+    );
+    const hashWasm = await import("hash-wasm");
+    const { isRejectedToken } = await import("./rejectedTokens");
+    vi.mocked(hashWasm.argon2Verify)
+      .mockRejectedValueOnce(new Error("Invalid hash"))
+      .mockRejectedValueOnce(new Error("Invalid hash"));
+
+    const invalid = {
+      isAuthorized: false,
+      statusCode: 401,
+      error: "Invalid token.",
+      reason: "invalidToken",
+    };
+    expect(await verifyTokenAndRateLimit("not-a-hash")).toEqual(invalid);
+    expect(await verifyTokenAndRateLimit("not-a-hash")).toEqual(invalid);
+
+    // Junk costs a regex reject, not a verification, so caching it would only
+    // spend a slot: it is refused again from scratch.
+    expect(hashWasm.argon2Verify).toHaveBeenCalledTimes(2);
+    expect(isRejectedToken("not-a-hash")).toBe(false);
+  });
+
   it("should accept valid token and add to verified tokens", async () => {
     mockArgon2VerifyResult = true;
     vi.resetModules();

server/verifyTokenAndRateLimit.ts

@@ -4,6 +4,11 @@ import path from "node:path";
 import debug from "debug";
 import { argon2Verify } from "hash-wasm";
 import { RateLimiterMemory } from "rate-limiter-flexible";
+import {
+  addRejectedToken,
+  isRejectedToken,
+  recordRejectedTokenCacheHit,
+} from "./rejectedTokens.ts";
 import { getSearchToken, hasSearchTokenFileChanged } from "./searchToken.ts";
 import { addVerifiedToken, isVerifiedToken } from "./verifiedTokens.ts";
 
@@ -146,18 +151,42 @@ export async function verifyTokenAndRateLimit(
   }
 
   if (!isVerifiedToken(token)) {
+    if (isRejectedToken(token)) {
+      // This process will never accept this token, so the refusal is final
+      // without paying for the verification again. The file-divergence check
+      // is skipped on purpose: this path has not paid for a verification, so
+      // its file read would dominate, and first rejections always run it,
+      // which is the case it was written for.
+      recordRejectedTokenCacheHit();
+
+      return {
+        isAuthorized: false,
+        statusCode: 401,
+        error: "Invalid token.",
+        reason: "invalidToken",
+      };
+    }
+
     let isValidToken = false;
+    let didComparisonRun = false;
 
     try {
       isValidToken = await argon2Verify({
         password: getSearchToken(),
         hash: token,
       });
+      didComparisonRun = true;
     } catch (error) {
       void error;
     }
 
     if (!isValidToken) {
+      // Only a comparison that ran to a result proves the token is dead. A
+      // throw means either the token file could not be read, and caching that
+      // would refuse a valid token for the rest of the process, or the hash
+      // is malformed, and caching junk would only spend a slot on a refusal
+      // that costs a regex check, not a verification.
+      if (didComparisonRun) addRejectedToken(token);
       reportTokenFileChangeOnce();
 
       return {