feat: add Jina search and read support - #32
Conversation
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (12)
📝 WalkthroughWalkthroughThis PR adds URL reading capability to askweb via the new Jina provider. It introduces ChangesRead Capability
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes The PR is substantial but follows a clear pattern: one new provider (Jina) plugged into existing abstractions. The logic is straightforward—URL validation, HTTP request building, response mapping—with no algorithmic complexity. Heterogeneity is low: all changes serve the same feature without conflicting patterns. Test coverage is comprehensive and mirrored closely to implementation (reduces surprise risk). The main review checkpoints are the Jina provider impl and the error handling paths in the CLI. Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/core/read.ts`:
- Around line 5-7: ReadUrlOptions currently declares provider?: string which
defers provider-name validation to runtime (create() throws); change provider to
a literal union of valid provider names (or reference an existing
ProviderName/ProviderType alias) instead of string — update the ReadUrlOptions
declaration (extending ReadOptions) to use that union, and ensure all callers
compile by using the union values; if a ProviderName type does not exist, add
one (e.g., type ProviderName = 's3' | 'gcs' | 'azure' | ...) and use provider?:
ProviderName so invalid provider names are caught at compile time.
In `@src/core/types.ts`:
- Around line 34-35: The public type for the fields links and images currently
allows either string[] or Record<string,string>, which forces callers to branch;
change the public contract in src/core/types.ts so links: string[] and images:
string[] (single normalized shape), and update each provider adapter/transform
code that populates those fields to convert any Record<string,string> shapes
into a flat array of URL strings (e.g., Object.values or mapping keys->values)
before assigning to the public model; keep internal/provider types if needed but
do not expose them on the public interface.
In `@test/unit/ai-tool.test.ts`:
- Around line 254-279: The test for readTool should not rely on the external
JINA_API_KEY environment variable; update the test to explicitly control
process.env.JINA_API_KEY around the call to readTool.execute (or clear it) so
headers are deterministic for mockGetJSON assertions. Locate the
describe('readTool') block and, in the it('reads a URL with Jina by default')
test, set process.env.JINA_API_KEY = undefined (or save and restore the previous
value) before invoking readTool.execute and restore the original value after the
assertion to ensure mockGetJSON.mock.calls[0][1] only contains Accept and
X-Respond-With headers.
In `@test/unit/read-command.test.ts`:
- Around line 56-57: Replace the unsafe type assertions: change the object cast
to use TypeScript's "satisfies ReadRunInput" so the value is type-checked rather
than asserted, remove the non-null assertion on readCommand.run and add a
runtime guard that throws if readCommand.run is undefined (e.g., if
(!readCommand.run) throw new Error('readCommand.run is not defined')), and
update the process.exit mock to have a properly typed callback that accepts an
optional code parameter (string | number | null) and throws a sentinel Error
(instead of using "as never") so the test fails via the thrown error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 89711c0d-1f4d-4e79-bd54-ccc6c5bd8313
📒 Files selected for processing (26)
README.mdpackage.jsonpackages/pi/extensions/askweb.tssrc/ai.tssrc/cli-args.tssrc/cli.tssrc/commands/read.tssrc/core/errors.tssrc/core/providers.tssrc/core/read.tssrc/core/resolve.tssrc/core/types.tssrc/index.tssrc/opencode.tssrc/providers/index.tssrc/providers/jina.tstest/index.test.tstest/unit/ai-tool.test.tstest/unit/all.test.tstest/unit/cli-args.test.tstest/unit/jina.test.tstest/unit/providers-command.test.tstest/unit/read-command.test.tstest/unit/read.test.tstest/unit/resolve-async.test.tstest/unit/resolve.test.ts
📜 Review details
🧰 Additional context used
📓 Path-based instructions (8)
{build.config.ts,package.json}
📄 CodeRabbit inference engine (AGENTS.md)
Keep
entriesandexportsaligned betweenbuild.config.tsandpackage.jsonfor build outputs
Files:
package.json
package.json
📄 CodeRabbit inference engine (AGENTS.md)
Default to minimal dependencies; only add HTTP/cache abstraction layers when provider adapters are being integrated
Files:
package.json
test/**/*.test.ts
📄 CodeRabbit inference engine (AGENTS.md)
test/**/*.test.ts: Mirror public behavior in tests, not implementation details
Do not make tests depend on external services; use mocks or fixtures instead
Files:
test/unit/resolve-async.test.tstest/unit/providers-command.test.tstest/unit/all.test.tstest/unit/read.test.tstest/unit/resolve.test.tstest/unit/ai-tool.test.tstest/unit/cli-args.test.tstest/unit/jina.test.tstest/unit/read-command.test.tstest/index.test.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{ts,tsx,js,jsx}: Package must be ESM-only with no CommonJS output
Do not introduce CommonJS compatibility shims; maintain ESM-only output
Files:
test/unit/resolve-async.test.tstest/unit/providers-command.test.tssrc/cli-args.tssrc/providers/index.tssrc/cli.tstest/unit/all.test.tstest/unit/read.test.tstest/unit/resolve.test.tssrc/core/read.tssrc/core/providers.tstest/unit/ai-tool.test.tssrc/core/errors.tstest/unit/cli-args.test.tssrc/core/types.tssrc/core/resolve.tstest/unit/jina.test.tssrc/ai.tssrc/opencode.tssrc/commands/read.tstest/unit/read-command.test.tspackages/pi/extensions/askweb.tstest/index.test.tssrc/providers/jina.tssrc/index.ts
**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Do not use
as any,@ts-ignore, or placeholder unsafe type assertions anywhere in the codebase
Files:
test/unit/resolve-async.test.tstest/unit/providers-command.test.tssrc/cli-args.tssrc/providers/index.tssrc/cli.tstest/unit/all.test.tstest/unit/read.test.tstest/unit/resolve.test.tssrc/core/read.tssrc/core/providers.tstest/unit/ai-tool.test.tssrc/core/errors.tstest/unit/cli-args.test.tssrc/core/types.tssrc/core/resolve.tstest/unit/jina.test.tssrc/ai.tssrc/opencode.tssrc/commands/read.tstest/unit/read-command.test.tspackages/pi/extensions/askweb.tstest/index.test.tssrc/providers/jina.tssrc/index.ts
src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
src/**/*.ts: Prefer normalized data models over provider-shaped raw objects in public API
Keep provider names and capability flags as literal unions rather than strings or enums
Do not leak provider-specific response formats into the public API; normalize all provider outputs
Do not couple CLI formatting logic with core data models; keep them separate
Files:
src/cli-args.tssrc/providers/index.tssrc/cli.tssrc/core/read.tssrc/core/providers.tssrc/core/errors.tssrc/core/types.tssrc/core/resolve.tssrc/ai.tssrc/opencode.tssrc/commands/read.tssrc/providers/jina.tssrc/index.ts
src/cli.ts
📄 CodeRabbit inference engine (AGENTS.md)
src/cli.ts: Extend CLI functionality usingcittysubcommands insrc/cli.ts; keep text and JSON output formats stable
CLI should be thin and delegate reusable functions tosrc/index.tsinstead of implementing network or core logic directly
CLI must support both human-readable and machine-readable JSON output formats
Do not add network code directly in the CLI; delegate HTTP requests to core modules fromsrc/index.ts
Files:
src/cli.ts
src/index.ts
📄 CodeRabbit inference engine (AGENTS.md)
Keep the public API surface small and explicit; manage all public exports through the barrel export at
src/index.ts
Files:
src/index.ts
🧠 Learnings (2)
📚 Learning: 2026-03-10T19:51:47.772Z
Learnt from: aeitwoen
Repo: oritwoen/websxa PR: 5
File: packages/opencode-websxa/package.json:39-40
Timestamp: 2026-03-10T19:51:47.772Z
Learning: In the oritwoen/websxa repo, using "latest" for devDependencies (e.g., obuild, typescript, vitest) is intentional and consistent across all workspace packages, including the root package.json. Do not treat this as a reproducibility issue during reviews. When reviewing package.json files across the monorepo, accept the use of "latest" for devDependencies and focus on other stability indicators (e.g., CI, lockfile integrity) instead.
Applied to files:
package.json
📚 Learning: 2026-03-18T13:22:47.573Z
Learnt from: oritwoen
Repo: oritwoen/websxa PR: 29
File: src/core/errors.ts:100-101
Timestamp: 2026-03-18T13:22:47.573Z
Learning: In src/core/errors.ts, keep the HAS_OFFSET_RE guard (hasTime && !HAS_OFFSET_RE.test(value)) in validateDateFilters as defense-in-depth. Do not flag this as dead/redundant code even though ISO_DATE_RE enforces offset now, because future regex changes could weaken or alter the guarantee. Treat this as intentional,Reviewed code that should not be removed or simplified during reviews.
Applied to files:
src/core/errors.ts
🔇 Additional comments (25)
README.md (1)
8-13: LGTM!Also applies to: 47-48, 83-100, 103-107, 111-123, 125-125, 129-129, 138-138, 146-146, 154-154, 162-162, 167-170, 179-179, 186-186, 191-191, 197-197, 227-227, 233-233, 252-252, 254-270, 285-285, 287-298
package.json (1)
4-4: LGTM!test/index.test.ts (1)
2-2: LGTM!Also applies to: 10-10, 15-15, 20-22
test/unit/ai-tool.test.ts (1)
18-20: LGTM!Also applies to: 65-65, 281-291
test/unit/all.test.ts (1)
53-53: LGTM!Also applies to: 377-377
test/unit/cli-args.test.ts (1)
7-7: LGTM!test/unit/jina.test.ts (1)
1-246: LGTM!test/unit/providers-command.test.ts (1)
14-14: LGTM!test/unit/read.test.ts (1)
1-35: LGTM!test/unit/resolve-async.test.ts (1)
11-11: LGTM!test/unit/resolve.test.ts (1)
5-5: LGTM!Also applies to: 37-37, 41-41, 53-53, 82-82
src/ai.ts (1)
6-7: LGTM!Also applies to: 12-12, 15-15, 42-61
src/opencode.ts (1)
7-7: LGTM!Also applies to: 13-13, 18-18, 35-47
packages/pi/extensions/askweb.ts (1)
7-8: LGTM!Also applies to: 35-42, 54-54, 56-57, 99-121, 125-125, 128-128, 135-135, 141-141, 233-285, 404-413, 432-442, 506-512, 531-540
src/cli-args.ts (1)
3-3: LGTM!src/cli.ts (1)
11-11: LGTM!Also applies to: 15-15
src/commands/read.ts (1)
1-146: LGTM!src/core/types.ts (1)
25-33: LGTM!Also applies to: 36-37, 39-46, 51-51, 65-65
src/core/errors.ts (1)
77-94: LGTM!src/index.ts (1)
7-9: LGTM!Also applies to: 18-20
src/core/read.ts (1)
1-4: LGTM!Also applies to: 9-25
src/core/providers.ts (1)
4-4: LGTM!src/core/resolve.ts (1)
8-8: LGTM!src/providers/jina.ts (1)
1-185: LGTM!src/providers/index.ts (1)
3-3: LGTM!
🤖 Augment PR SummarySummary: This PR adds Jina support as a new built-in provider and introduces first-class URL “read” capability alongside existing query “search”. Changes:
Technical Notes: Jina search requires Bearer auth via 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Requires human review: This PR introduces a new search provider (Jina) and a new read capability spanning core logic, CLI, AI SDK tools, Pi extension, OpenCode plugin, and tests, which represents a substantial feature addition with broad blast radius and multiple integration points requiring careful human review to...
Re-trigger cubic
There was a problem hiding this comment.
8 issues found across 26 files
Confidence score: 3/5
- There is moderate user-impact risk because
src/providers/jina.tshas functional gaps: using the wrong read-format header and not validating Jina’s application-levelresponse.code/response.statuscan silently return wrong content or hide provider-side errors. src/commands/read.tsshould sanitize fetched page text before printing; without this, terminal escape sequences from remote content can pass through directly to users.- Several findings look non-blocking but still worth fixing soon (
src/commands/read.tsprovider listing UX,src/core/read.tserror-ordering,test/unit/ai-tool.test.tsenv sensitivity, andREADME.mdtyping/signature mismatch), which keeps this from feeling like a low-risk merge. - Pay close attention to
src/providers/jina.ts,src/commands/read.ts, andsrc/core/read.ts- provider routing/error handling and terminal output safety are the main regression points.
Architecture diagram
sequenceDiagram
participant CLI as CLI / Commands
participant TUI as Pi Extension / OpenCode
participant AI as AI SDK Tool
participant Read as core/read.ts
participant Registry as core/registry.ts
participant JinaProv as Jina Provider
participant JinaAPI as s.jina.ai / r.jina.ai
participant SearchProv as Search Providers
participant Errors as core/errors.ts
Note over CLI,JinaAPI: NEW: Read flow (URL-to-content)
CLI->>CLI: askweb read <url> --format markdown
CLI->>Read: readUrl(url, { provider, format, maxTokens })
Read->>Registry: create(providerName)
Registry-->>Read: JinaProvider instance
Read->>JinaProv: provider.read(url, readOptions)
alt Read without API key (basic)
JinaProv->>JinaAPI: GET r.jina.ai/url with Accept: application/json
else Read with JINA_API_KEY
JinaProv->>JinaAPI: GET r.jina.ai/url with Authorization: Bearer {key}
end
JinaAPI-->>JinaProv: { code, data: { title, content, text, ... } }
JinaProv->>JinaProv: mapReadResult() → normalized ReadResult
JinaProv-->>Read: ReadResult { url, title, content, ... }
Read-->>CLI: ReadResult
CLI->>CLI: Output formatted result or JSON
Note over TUI,AI: NEW: Tool/extension integration
alt Pi extension tool call
TUI->>TUI: askweb_read.execute({ url, provider, format, ... })
TUI->>Read: askweb.readUrl(url, options)
Read-->>TUI: ReadResult
TUI->>TUI: formatReadResult() with header
TUI-->>TUI: Return agent tool result
else OpenCode plugin call
TUI->>TUI: askweb_read.execute({ url, provider, format })
TUI->>Read: readUrl(url, { provider, format, maxTokens })
Read-->>TUI: ReadResult
TUI->>TUI: encode(ReadResult)
else AI SDK tool call
AI->>AI: readTool.execute({ url, format })
alt Empty URL
AI->>Errors: throw EmptyUrlError
end
AI->>Read: readUrl(url, { provider: 'jina', format, ... })
Read-->>AI: ReadResult
AI-->>AI: Return ReadResult
end
Note over CLI,JinaAPI: CHANGED: Jina search flow
CLI->>CLI: askweb search <query> --provider jina
CLI->>Registry: create('jina', { apiKey: env.JINA_API_KEY })
Registry-->>CLI: JinaProvider
CLI->>JinaProv: search(query, { maxResults, category, includeDomains })
alt Missing API key
JinaProv->>Errors: throw AuthError
end
JinaProv->>JinaAPI: GET s.jina.ai/search?q=... with Bearer auth
JinaAPI-->>JinaProv: Search results
JinaProv->>JinaProv: mapSearchResult() → normalized SearchResult
JinaProv-->>CLI: SearchResult[]
Note over Registry,Errors: CHANGED: Registry with read capability check
Registry->>Registry: provider.read() optional method
alt Provider implements read
Read->>JinaProv: read() → success
else Provider has no read method
Read->>Errors: throw ReadNotSupportedError
end
Note over Registry: NEW: ProviderConfig.readBaseURL and deriveReadBaseURL()
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
1 issue found across 10 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/core/read.ts">
<violation number="1" location="src/core/read.ts:6">
P2: This adds another hardcoded read-provider list. Reuse the shared export instead of keeping three copies. If the next reader only updates `src/core/read.ts`, the AI and opencode tools still reject it.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Requires human review: Auto-approval blocked by 1 unresolved issue from previous reviews.
Re-trigger cubic
1 participant
Adds Jina as search provider through
s.jina.ai, wired into registry, default detection, CLI/tool surfaces, and docs. Also adds first read capability withreadUrl+askweb read/askweb_read, backed by Jina Reader atr.jina.ai- URL to content stays separate from query to results, so scope doesn't get muddy.