feat: add Oathe behavioral security scan integration#562
feat: add Oathe behavioral security scan integration#562liamkauffman wants to merge 7 commits intoopenclaw:mainfrom
Conversation
- countPublicSkills returns 0 instead of full table scan on fresh deploy - adjustGlobalPublicSkillsCount logs warning instead of silent swallow - Remove unused countPublicSkillsForGlobalStats import - Tests: moderationFlags malware exclusion, undefined moderationStatus - Tests: mock validates query builder callback filters softDeletedAt - Docs: ordering requirement JSDoc, OCC contention trade-off comment
Add automatic security scanning via Oathe API: - Submit skills for audit on publish (fire-and-forget) - Cron polls every 10min for pending results, re-submits after 1hr, times out at 24hr - Display Oathe verdict, score, dimensions alongside VT and LLM analysis - New oatheAnalysis field on skillVersions schema Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add scan-result-icon-oathe, version-scan-icon-oathe (--seafoam), and scan-status-unknown (gray fallback) CSS classes referenced by SkillSecurityScanResults.tsx. Revert Footer.tsx "Powered by Convex" removal to keep PR scoped to Oathe integration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change reportUrl from cloud.oathe.ai/api/skill/{slug}/latest (JSON)
to oathe.ai/report/{slug} (rendered report page). Remove unused
apiUrl parameter from mapReportToAnalysis.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@liamkauffman is attempting to deploy a commit to the Amantus Machina Team on Vercel. A member of the Team first needs to authorize it. |
…alidation - Add submittedAt field to oatheAnalysis schema so cron escalation logic (force_rescan after 1h, timeout after 24h) works correctly instead of resetting pendingSince every cycle via checkedAt - Add 17 unit tests for scoreToRating, verdictToStatus, mapReportToAnalysis - Add protocol validation (isSafeUrl) for reportUrl href rendering - Export pure functions via __test for testability Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d732831dbd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| versionId, | ||
| oatheAnalysis: { | ||
| status: 'pending', | ||
| submittedAt: pendingSince, |
There was a problem hiding this comment.
After the 1h retry path runs, submittedAt is written back as the old pendingSince value, but pendingAge is computed from that same field on every cron run. In fetchPendingOatheResults, this keeps the age permanently > ONE_HOUR_MS, so the skill is force-resubmitted every 10-minute cycle instead of waiting, which can spam Oathe and trigger avoidable rate limiting for any long-running audit.
Useful? React with 👍 / 👎.
| const allSkills = await ctx.db | ||
| .query('skills') | ||
| .withIndex('by_active_updated', (q) => q.eq('softDeletedAt', undefined)) | ||
| .order('desc') | ||
| .take(poolSize) |
There was a problem hiding this comment.
getSkillsPendingOatheInternal only samples the most recently updated skills (by_active_updated descending) and caps the pool to 1000 records. If enough newer skill updates happen, older pending scans fall outside that window and will never be returned to the cron poller, so they can remain stuck in pending indefinitely without timeout/resubmit handling.
Useful? React with 👍 / 👎.
Greptile SummaryThis PR integrates Oathe as a third behavioral security scanner alongside VirusTotal and OpenClaw's LLM analysis. The integration is well-structured and follows the existing VT/LLM patterns closely — schema, mutations, cron, and UI rendering are all consistent with how the other scanners are wired. Notable issues found:
Confidence Score: 3/5
Last reviewed commit: d732831 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
convex/oathe.ts
Outdated
| return | ||
| } | ||
|
|
||
| const skillUrl = `https://clawhub.ai/${skill.slug}` |
There was a problem hiding this comment.
Hardcoded production URL will misfire in dev/staging
https://clawhub.ai is hardcoded as the skill base URL in two places (notifyOathe on line 166 and the cron re-submit on line 304). In any non-production Convex deployment (dev, staging, preview), Oathe will scan the live production URL instead of the environment-appropriate one. This means:
- Skills not yet published to production will get "not found" or stale scan results from Oathe.
- Dev/staging publish events will silently inflate Oathe's scan queue with production URLs.
The Oathe API URL is already gated on OATHE_API_URL, so adding a companion env var for the base URL would be consistent. For example:
const skillBaseUrl = process.env.CLAWHUB_BASE_URL ?? 'https://clawhub.ai'
const skillUrl = `${skillBaseUrl}/${skill.slug}`The same fix is needed at line 304 in the cron re-submit path.
Prompt To Fix With AI
This is a comment left during a code review.
Path: convex/oathe.ts
Line: 166
Comment:
**Hardcoded production URL will misfire in dev/staging**
`https://clawhub.ai` is hardcoded as the skill base URL in two places (`notifyOathe` on line 166 and the cron re-submit on line 304). In any non-production Convex deployment (dev, staging, preview), Oathe will scan the *live production* URL instead of the environment-appropriate one. This means:
- Skills not yet published to production will get "not found" or stale scan results from Oathe.
- Dev/staging publish events will silently inflate Oathe's scan queue with production URLs.
The Oathe API URL is already gated on `OATHE_API_URL`, so adding a companion env var for the base URL would be consistent. For example:
```typescript
const skillBaseUrl = process.env.CLAWHUB_BASE_URL ?? 'https://clawhub.ai'
const skillUrl = `${skillBaseUrl}/${skill.slug}`
```
The same fix is needed at line 304 in the cron re-submit path.
How can I resolve this? If you propose a fix, please make it concise.
convex/skills.ts
Outdated
| const poolSize = Math.min(Math.max(limit * 20, 200), 1000) | ||
| const allSkills = await ctx.db | ||
| .query('skills') | ||
| .withIndex('by_active_updated', (q) => q.eq('softDeletedAt', undefined)) | ||
| .order('desc') | ||
| .take(poolSize) | ||
|
|
||
| const results: Array<{ | ||
| skillId: Id<'skills'> | ||
| versionId: Id<'skillVersions'> | ||
| slug: string | ||
| pendingSince: number | ||
| }> = [] | ||
|
|
||
| for (const skill of allSkills) { | ||
| if (results.length >= limit) break | ||
| if (!skill.latestVersionId) continue | ||
|
|
||
| const version = await ctx.db.get(skill.latestVersionId) |
There was a problem hiding this comment.
Sequential N+1 reads up to 1000 documents per cron run
With limit=50 (the default batchSize from the cron), poolSize evaluates to Math.min(Math.max(50 * 20, 200), 1000) = 1000. The loop then does a sequential ctx.db.get(skill.latestVersionId) for each of the 1000 fetched skills, resulting in up to 2,000 document reads every 10 minutes just to find 50 pending skills. The reads are sequential (not Promise.all), so this adds latency on top of the read cost.
This is especially wasteful in the steady state where most skills don't have a pending Oathe scan — the full 1,000-skill pool will be scanned without finding anything to return.
Two suggestions:
- Short-term: Replace the sequential loop with parallel reads using
Promise.allto reduce latency, and consider a smallerpoolSizemultiplier now that the cron runs frequently (every 10 min). - Longer-term: Store pending Oathe version IDs in a lightweight auxiliary table (e.g.,
oathePendingQueue) and index on it, avoiding the skills scan entirely. The existing VT integration likely has a comparable pattern worth following.
Prompt To Fix With AI
This is a comment left during a code review.
Path: convex/skills.ts
Line: 2694-2712
Comment:
**Sequential N+1 reads up to 1000 documents per cron run**
With `limit=50` (the default batchSize from the cron), `poolSize` evaluates to `Math.min(Math.max(50 * 20, 200), 1000) = 1000`. The loop then does a sequential `ctx.db.get(skill.latestVersionId)` for each of the 1000 fetched skills, resulting in up to **2,000 document reads every 10 minutes** just to find 50 pending skills. The reads are sequential (not `Promise.all`), so this adds latency on top of the read cost.
This is especially wasteful in the steady state where most skills don't have a pending Oathe scan — the full 1,000-skill pool will be scanned without finding anything to return.
Two suggestions:
1. **Short-term**: Replace the sequential loop with parallel reads using `Promise.all` to reduce latency, and consider a smaller `poolSize` multiplier now that the cron runs frequently (every 10 min).
2. **Longer-term**: Store pending Oathe version IDs in a lightweight auxiliary table (e.g., `oathePendingQueue`) and index on it, avoiding the skills scan entirely. The existing VT integration likely has a comparable pattern worth following.
How can I resolve this? If you propose a fix, please make it concise.
Additional Comments (1)
return url.startsWith('https://') || url.startsWith('http://')The Prompt To Fix With AIThis is a comment left during a code review.
Path: src/components/SkillSecurityScanResults.tsx
Line: 1083-1085
Comment:
**`isSafeUrl` permits `http://` links**
```typescript
return url.startsWith('https://') || url.startsWith('http://')
```
The `reportUrl` is always written server-side as `https://oathe.ai/report/${slug}`, so `http://` is never reached in practice today. However, the permissive check means any future code path that writes an `http://` value would render as a mixed-content anchor on the HTTPS site without any warning. Restricting to `https://` only would be a safer default:
```suggestion
function isSafeUrl(url: string): boolean {
return url.startsWith('https://')
}
```
How can I resolve this? If you propose a fix, please make it concise. |
- Add rescanAt field to prevent force-rescan from firing every cron cycle (P1) - Parallelize version reads with Promise.all in getSkillsPendingOatheInternal - Replace hardcoded https://clawhub.ai with SITE_URL env var in both paths - Restrict isSafeUrl to https:// only Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e1ff4904a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
convex/oathe.ts
Outdated
| } | ||
|
|
||
| const siteUrl = (process.env.SITE_URL ?? 'https://clawhub.ai').replace(/\/+$/, '') | ||
| const skillUrl = `${siteUrl}/${skill.slug}` |
There was a problem hiding this comment.
notifyOathe constructs skill_url as ${siteUrl}/${skill.slug}, but skill detail routes are defined as /$owner/$slug (see src/routes/$owner/$slug.tsx) and there is no single-segment skill route under src/routes. When Oathe resolves the submitted URL, this can target a non-skill page/404, so behavioral scans run against the wrong target (and the same pattern is reused in the force-rescan submit path), which undermines scan correctness for published skills.
Useful? React with 👍 / 👎.
The Oathe API requires two-segment skill URLs (owner/slug) but we were sending one-segment URLs (slug only), causing 400s on submit and silent 404s on poll. Look up owner handle and skip submission if missing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1f58ea499
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (!ownerHandle) { | ||
| console.warn(`[oathe] Skipping ${skill.slug}: owner has no handle`) | ||
| return |
There was a problem hiding this comment.
notifyOathe exits early when ownerHandle is empty, but user handles are optional and skill URLs elsewhere already fall back to other owner identifiers, so this path silently skips behavioral scanning for valid published skills from users without a populated handle. In that scenario no Oathe request is sent and no oatheAnalysis state is written, leaving those versions permanently unscanned even when OATHE_API_URL is configured.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds Oathe behavioral security scanning as a third scanner alongside VirusTotal and OpenClaw's LLM analysis. Oathe runs skills in a sandboxed VM and monitors actual runtime behavior (filesystem, network, process activity), catching attack vectors that static analysis misses.
OATHE_API_URLenv var — zero impact when unsetHow it works
What the UI looks like
All three scanners together (VT + OpenClaw + Oathe)
Oathe "Safe" result with expanded dimensions
Oathe "Caution" result with behavioral findings
Pending state
Files changed
convex/oathe.tsconvex/schema.tsoatheAnalysisfield onskillVersionsconvex/crons.tsconvex/skills.tsconvex/lib/skillPublish.tsnotifyOatheon publishsrc/components/SkillSecurityScanResults.tsxsrc/components/SkillHeader.tsxoatheAnalysisto SecurityScanResultssrc/components/SkillVersionsPanel.tsxoatheAnalysisto badge variantsrc/styles.css.scan-result-icon-oathe,.version-scan-icon-oathe,.scan-status-unknown.github/workflows/secret-scan.ymlconvex/lib/globalStats.tsconvex/skills.countPublicSkills.test.tsOpt-in design
Set
OATHE_API_URLin your Convex environment to enable. When unset:notifyOatheexits immediately with a log lineNo new dependencies. No impact to existing VT or OpenClaw scanning. Fully env-var-gated — can be enabled per-environment when ready.
Strategic alignment
Roadmap alignment
OpenClaw's Trust & Security page identifies supply chain security as critical: "Skills are code that runs in your agent's context — supply chain security is critical." The published security program's Phase 3 (code review + dynamic testing) covers behavioral analysis — an area where existing scanners have a known gap. VirusTotal catches known malware signatures, and OpenClaw's LLM scanner evaluates skill bundles statically, but neither observes actual runtime behavior: exfiltration over external URLs, social-engineered payloads, or adversarial SKILL.md instructions that don't surface in source code.
The Security Enhancements RFC discusses sandboxing as a defensive goal. This PR contributes a community-built integration for exactly that — a behavioral sandbox that executes skills in an isolated VM and monitors filesystem, network, and process activity.
Complementary PRs
/api/v1/skills/{slug}/scanendpoint with merged security snapshot. Oathe can be added toSkillSecuritySnapshot.scannersas a follow-up once both PRs land.The Oathe team has confirmed support (see issue comment).
Test plan
bun run lint— 0 warnings, 0 errorsbun run build— client + SSR + Nitro all passbun run test— 375 tests pass (39 pre-existinghtml-encoding-snifferESM errors unrelated to this PR)fetchPendingOatheResults→ resolved with real Oathe API data (Caution 72/100)oathe.ai/report/{slug}, not raw API JSONOATHE_API_URLis not set (all Oathe code is skipped)AI disclosure
This PR was developed with AI assistance (Claude). All code was reviewed, tested locally with seeded and live API data, and verified against the existing VT/OpenClaw scan UI patterns.
Closes #288