feat: add Firecrawl provider

[oritwoen]

Adds Firecrawl as a search and read provider. Search through /v2/search, scrape through /v2/scrape. Env var: FIRECRAWL_API_KEY. Supports domain filters and news category. Read returns markdown by default.

Also added a provider addition checklist to AGENTS.md since there are 7 files to touch when adding a new provider and it's easy to miss one.

[cubic-dev-ai]

4 issues found across 9 files

Confidence score: 3/5

• There is concrete user-impact risk in src/providers/firecrawl.ts: read() can return empty content for non-markdown requests because content is still sourced from markdown, which can surface as blank output in CLI/API consumers.
• src/providers/firecrawl.ts also has two medium-severity integration mismatches (format: 'text' passed to Firecrawl and news sources mapped through data.web), both of which can cause failed reads or empty results for valid user requests.
• Given multiple high-confidence issues at severity 6–7/10 in request/response handling paths, this looks like moderate merge risk rather than a safe immediate merge.
• Pay close attention to src/providers/firecrawl.ts and AGENTS.md - provider behavior may fail for text/news flows, and the checklist wording could steer future implementations away from the intended capability model.

Architecture diagram

sequenceDiagram
    participant P as Pi Extension (askweb.ts)
    participant R as Resolver (resolve.ts)
    participant F as Firecrawl Provider (firecrawl.ts)
    participant CL as HTTP Client (client.ts)
    participant FC as Firecrawl API (api.firecrawl.dev)
    participant Reg as Registry (registry.ts)
    participant Read as Read Provider (read.ts)

    Note over P,FC: Runtime Flow for Firecrawl Provider

    P->>R: resolve("firecrawl")
    R->>R: lookup FIRECRAWL_API_KEY env var
    R-->>P: provider configuration

    P->>Reg: create("firecrawl", config)

    Reg->>F: new FirecrawlProvider(config)
    Note over F: Checks apiKey, throws AuthError if missing
    F-->>Reg: provider instance

    alt Search Flow
        P->>F: search(query, options?)
        F->>F: clampMaxResults(options?.maxResults)
        alt includeDomains/excludeDomains provided
            F->>F: add domain filters to body
        end
        alt category === "news"
            F->>F: set sources: ["news"]
        end
        F->>CL: postJSON(url="/v2/search", body, authHeaders)
        CL->>FC: POST /v2/search (Authorization: Bearer <key>)
        FC-->>CL: FirecrawlSearchResponse
        CL-->>F: parsed response
        alt success == true
            F->>F: map each result (markdown → text field)
            F-->>P: SearchResult[]
        else success == false
            F->>F: throw Error ("search failed")
            F-->>P: normalized error
        end
    end

    alt Read Flow
        P->>Read: resolve read provider
        Read->>Read: firecrawl in readProviderNames list
        Read-->>P: read provider resolved

        P->>F: read(url, options?)
        F->>F: set formats: [options?.format] (default "markdown")
        opt options?.timeout provided
            F->>F: convert seconds to milliseconds
        end
        F->>CL: postJSON(url="/v2/scrape", body, authHeaders)
        CL->>FC: POST /v2/scrape (Authorization: Bearer <key>)
        FC-->>CL: FirecrawlScrapeResponse
        CL-->>F: parsed response
        alt success == true
            F->>F: extract markdown, html, metadata
            F-->>P: ReadResult with content, title, description, image
        else success == false
            F->>F: throw Error ("scrape failed")
            F-->>P: normalized error
        end
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 6 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Requires human review: Adding a new provider touches core registration, resolution, Pi tool descriptions, and environment variable mappings—changes to business logic and integration points that require a human reviewer to verify correctness and security.

Re-trigger cubic</sub>