feat: geofilter customizer tab + PUT /api/config/geo-filter#736
feat: geofilter customizer tab + PUT /api/config/geo-filter#736efiten wants to merge 20 commits into
Conversation
Kpa-clawbot
left a comment
There was a problem hiding this comment.
Security Review — PUT /api/config/geo-filter
Reviewed with a security-focused lens on the config write endpoint. The overall design is solid — API key gating, atomic file writes, input shape validation, clear separation of read-only vs write-enabled UI. Good work. But there are a few issues that need addressing before merge.
Must-fix
1. Data race on s.cfg.GeoFilter (routes.go)
The PUT handler writes s.cfg.GeoFilter = gf with no synchronization. The GET handler and any ingest-time filtering read s.cfg.GeoFilter concurrently from other goroutines. This is a textbook data race — Go's race detector will flag it. Wrap config mutations in a mutex (or use atomic.Pointer if you want lock-free reads).
2. No request body size limit (routes.go:~L440)
json.NewDecoder(r.Body).Decode(&body) reads unbounded input. An attacker with a valid API key (or a compromised key) can send a multi-GB JSON body and exhaust server memory. Wrap with http.MaxBytesReader:
r.Body = http.MaxBytesReader(w, r.Body, 1<<20) // 1 MB3. No coordinate range validation (routes.go)
Polygon points are stored as-is — no check that lat ∈ [-90, 90] and lon ∈ [-180, 180]. NaN, Infinity, or wildly out-of-range values would be persisted to config.json and could cause downstream issues (point-in-polygon math, Leaflet rendering). Validate each point after decoding.
Should-fix
4. writeEnabled leaks API key presence to unauthenticated callers (routes.go:~L429)
The GET endpoint (no auth required) returns writeEnabled: true/false, which tells any anonymous visitor whether the server has a strong API key configured. This is a minor information leak — consider removing it from the unauthenticated response and having the UI infer editability from a separate authenticated probe, or accept the risk with a comment explaining the trade-off.
5. No upper bound on polygon point count
A polygon with 100,000 points is technically valid per the current checks. Consider a reasonable cap (e.g., 1000 points) to bound storage and computation.
Nits (non-blocking)
SaveGeoFiltererror messages include filesystem paths (config.json not found in /app/data). These get logged server-side (fine) but also returned to the client viawriteError. Consider generic client-facing messages.- The
_commentfield inconfig.example.jsonis a nice touch for discoverability.
What's done well
requireAPIKeymiddleware on the PUT — correct layer for authzIsWeakAPIKeycheck forwriteEnabled— defense in depth- Atomic write via temp + rename — prevents partial writes on crash
- Polygon minimum 3 points validation
- Comprehensive test coverage: auth rejection, save/clear/validation, disk persistence
- UI correctly gates edit controls on server-reported
writeEnabled - Clean separation: standalone builder for offline use, customizer tab for live editing
Items 1–3 are must-fix before merge. The rest can be addressed in follow-ups.
- Fix data race on s.cfg.GeoFilter: add cfgMu RWMutex with getGeoFilter/ setGeoFilter accessors used in all handler goroutines - Add 1 MB MaxBytesReader cap on PUT /api/config/geo-filter request body - Validate polygon coordinate ranges (lat ∈ [-90,90], lon ∈ [-180,180]) and reject NaN/Inf values - Cap polygon at 1000 points to bound storage and computation - Document writeEnabled information-leak trade-off with a comment - Add tests for coordinate range rejection and oversized polygon rejection Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
All must-fix and should-fix items from the security review have been addressed in commit Must-fix — resolved:
Should-fix — resolved: Nit (filesystem path in error response) — left as-is; the path leaks only to authenticated callers (PUT requires Build was also broken by a missing closing |
Final Re-Review — NEEDS REBASEFollow-up to the security review from 2026-04-15. Verifying @efiten's claimed fixes in 🔒 djb — Security VerificationMust-fix #1 (data race): ✅ Fixed. Must-fix #2 (unbounded body): ✅ Fixed. Must-fix #3 (coordinate validation): ✅ Fixed. NaN/Inf/range checks at L487-490, lat ∈ [-90,90], lon ∈ [-180,180]. Should-fix #4 (writeEnabled leak): ✅ Acknowledged. Comment at L448-450 documents the risk-accepted trade-off. Reasonable. Should-fix #5 (polygon cap): ✅ Fixed. 1000-point cap at L484. Nit (error paths leak): ✅ Fixed. All five original findings verified as resolved. 📐 dijkstra — Correctness of the Mutex DisciplineThe RWMutex usage is correct: reads take RLock, writes take Lock. The invariant "in-memory ≥ disk" holds because One observation: 🆕 New Findings
VerdictReady after rebase. All must-fix and should-fix items verified as resolved. No new blocking issues. The PR is currently |
|
@efiten can you please resurrect this PR - either rebase it and let's get it merged or if it's stale let's close it as such. |
…bot#669 M3) Backend: - Add PUT /api/config/geo-filter (requires X-API-Key) — saves geo_filter back to config.json atomically and updates in-memory config immediately, no restart needed - Add SaveGeoFilter() to config.go: reads config as raw map (preserving _comment fields), updates geo_filter key, writes back via temp+rename - Add writeEnabled field to GET /api/config/geo-filter response so the frontend can gate editing controls on server write capability - Add Server.configDir field; wired from -config-dir flag in main.go - Tests: TestPutConfigGeoFilter (4 cases) + TestSaveGeoFilter (3 cases) Frontend: - Add GeoFilter tab (🗺️) to the customizer between Display and Export - Tab shows current polygon on a Leaflet map (read-only for all users) - Editing controls (undo, clear, buffer km, API key input, save/remove) are only revealed when the server reports writeEnabled=true — i.e. the deployment has a write-capable apiKey configured. Public instances see a read-only polygon view. - Save calls PUT /api/config/geo-filter; Remove clears the filter - Map is destroyed on tab switch and panel close to avoid Leaflet leaks Docs: - Add docs/user-guide/geofilter.md (full guide: config, customizer, builder, prune script, API) - Update configuration.md and customization.md with geo_filter section - Update config.example.json _comment to mention the Customizer tab Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clicking the small inline map in the customizer GeoFilter tab now opens a full-screen modal (92vw × 86vh) with Undo/Clear/Done/Cancel controls. The inline map becomes a read-only preview. Both maps and the standalone geofilter-builder.html now use CartoDB Positron (light) instead of dark. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix data race on s.cfg.GeoFilter: add cfgMu RWMutex with getGeoFilter/ setGeoFilter accessors used in all handler goroutines - Add 1 MB MaxBytesReader cap on PUT /api/config/geo-filter request body - Validate polygon coordinate ranges (lat ∈ [-90,90], lon ∈ [-180,180]) and reject NaN/Inf values - Cap polygon at 1000 points to bound storage and computation - Document writeEnabled information-leak trade-off with a comment - Add tests for coordinate range rejection and oversized polygon rejection Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
e12dc6d to
5719b9e
Compare
- config.go: keep SaveGeoFilter (PR Kpa-clawbot#736) + add upstream observer blacklist helpers and AnalyticsConfig/recompute-interval methods (Kpa-clawbot#1240) - routes.go: add upstream foreign-node pass-through (Kpa-clawbot#730) before the NodePassesGeoFilter check; keep gf local variable from PR - config.example.json: merge geo_filter _comment (mention both Customizer and Builder) + add foreignAdverts block from upstream (Kpa-clawbot#730) - geofilter-builder.html: take upstream dark theme + Save/Load/Download button styles (Kpa-clawbot#736 r2) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Head branch was pushed to by a user without write access
… 0.83px LCD clip (Kpa-clawbot#1251) Red: master CI run https://github.com/Kpa-clawbot/CoreScope/actions/runs/25995768081 already fails on `test-e2e-playwright.js` `Kpa-clawbot#1221 LCD clipped on right (right=375.828125, vw=375)`. No new test commit — the existing E2E assertion is the gate. **Root cause.** PR Kpa-clawbot#1222's mobile rule set `.vcr-bar { padding: 4px 8px }`. The flex row holds three `flex-shrink: 0` children (controls + scope-btns + lcd) and one `flex: 1 1 0` absorber (`.vcr-timeline-container`, `min-width: 40px`). At 375px viewport the absorber hits its floor, so the intrinsic widths of the shrink-frozen children spill 0.83px past the padding box. **Fix.** Drop horizontal padding 8px → 4px inside the `@media (max-width: 640px)` block. That's 8px of new slack — order of magnitude above the 0.83px clip — keeping LCD's `getBoundingClientRect().right ≤ 375`. Desktop layout untouched (rule is mobile-scoped). VCR/feed overlap (Kpa-clawbot#1206/Kpa-clawbot#1213) not reintroduced because `--vcr-bar-height` is JS-measured by the ResizeObserver, not pinned in CSS. Fixes Kpa-clawbot#1250 Co-authored-by: openclaw-bot <bot@openclaw.local>
Kpa-clawbot#1252) Failing test commit: `bdb4eefb` (added in Kpa-clawbot#1189 R1) — original CI failure: https://github.com/Kpa-clawbot/CoreScope/actions/runs/25995819598 Fixes Kpa-clawbot#1249. ## Root cause Two independent bugs surfaced by the same E2E test: 1. **Fixture join broken.** `scripts/capture-fixture.sh` wrote the text observer hash into `observations.observer_idx`, but the v3 join in `cmd/server` is `observers.rowid = observations.observer_idx`. The join silently nulled out `observer_id` / `observer_iata` for every packet. 2. **Mobile clipping.** `.col-observer` had `data-priority=3` (hides at ≤1024px) and was in the narrow-viewport `defaultHidden` list, so at 375px the cell collapsed to `display:none` and `.badge-iata` had a 0×0 box. ## Changes - `test-fixtures/e2e-fixture.db`: remap `observer_idx` text hash → integer rowid (500/500 rows resolved). - `scripts/capture-fixture.sh`: build an `observer_id → rowid` map before insert; skip rows whose observer isn't in the fixture. Comment explains the trap. - `public/packets.js`: bump `.col-observer` priority `3 → 1` and drop `observer` from narrow-viewport `defaultHidden`. ## Verification All three sub-tests in `test-observer-iata-1188-e2e.js` pass locally against the freshened fixture. `curl /api/packets?limit=5` returns real IATA codes (OAK / MRY / SFO) instead of empty strings. Co-authored-by: OpenClaw Bot <bot@openclaw.local>
…1.25px clip (Kpa-clawbot#1255) Fixes Kpa-clawbot#1254. Master CI Playwright fail-fast on every push since Kpa-clawbot#1252: ``` ❌ Mobile viewport (375px): observer IATA badge stays visible — not clipped: .badge-iata right edge 376.25 exceeds 375px viewport ``` ## Root cause After Kpa-clawbot#1252 unhid `.col-observer` at narrow widths so the IATA pill from Kpa-clawbot#1188 renders on mobile, at 375px the cell padding + truncated observer name (10 chars in grouped rows) + `.badge-iata` pill (`padding: 1px 5px` + `margin-left: 4px`) sums to ~376.25px — overflowing the viewport by 1.25px. Same class of failure as Kpa-clawbot#1250/Kpa-clawbot#1251 (VCR LCD-clip). ## Fix `public/style.css` — inside the existing `@media (max-width: 640px)` block, shrink `.badge-iata` `padding: 1px 5px → 1px 3px` and `margin-left: 4px → 2px`. Reclaims ~6px horizontally, well clear of the 1.25px overflow. Desktop (≥641px) styling untouched. ## TDD The failing E2E sub-test in `test-observer-iata-1188-e2e.js` (added in Kpa-clawbot#1189 R1) IS the red. Mutation verified locally: | Variant | Result | |--------------------|--------| | WITHOUT this fix | ❌ `.badge-iata right edge 376.25 exceeds 375px viewport` | | WITH this fix | ✅ all 3 sub-tests pass | ## Local verification ``` $ go build -o /tmp/corescope-server ./cmd/server $ /tmp/corescope-server -port 13581 -db test-fixtures/e2e-fixture.db -public public & $ CHROMIUM_PATH=/usr/bin/chromium BASE_URL=http://localhost:13581 \ node test-observer-iata-1188-e2e.js Running observer-IATA E2E tests against http://localhost:13581 ✅ Packets table renders an IATA badge in an observer cell ✅ Filter grammar: observer_iata == "<code>" narrows the table ✅ Mobile viewport (375px): observer IATA badge stays visible — not clipped All observer-IATA E2E tests passed. ``` ## Constraints honored - All colors via existing CSS variables (no theming illusions; only `padding` / `margin-left` change inside `@media (max-width: 640px)`). - No JS changes. - Desktop badge display unaffected (selector scoped to narrow viewport). - `config.example.json`: no config field added. - PII preflight: clean. Co-authored-by: OpenClaw Bot <bot@openclaw.local>
Kpa-clawbot
left a comment
There was a problem hiding this comment.
Consolidated review — PR #736
Adversarial review. Verdict: NEEDS-WORK — security regression at P0.
MUST-FIX
-
🔴 Security: file mode downgrade on every save.
SaveGeoFilterdoesos.WriteFile(tmp, out, 0644)thenos.Rename. Operators commonly chmodconfig.jsonto0600because it containsapiKey(the very same key the PUT endpoint authenticates with). Every successful PUT will silently re-create the file world-readable. Fix:os.Stat(configPath)to capture original mode, apply viaos.Chmod(tmp, st.Mode())(orOpenFilewith that mode) before rename. -
bufferKmis not validated. PUT validates every polygon point for NaN/Inf/range but letsbufferKmthrough unchecked.{"polygon":[…],"bufferKm":-1e9}orNaNwrites straight to disk and intoGeoFilterConfig. Negative buffer inverts geometric semantics inNodePassesGeoFilter; NaN poisons comparisons. Fix: rejectIsNaN/IsInf, require>= 0, impose a sane upper bound (e.g., ≤ 20000 km — half-Earth circumference is the only meaningful ceiling). -
Concurrent-PUT read-modify-write race on disk.
cfgMuonly guards the in-memory pointer.SaveGeoFilterreadsconfig.json, mutates the raw map, writes tmp, renames — with no serialization. Two concurrent authenticated PUTs can (a) collide on the sharedconfigPath+".tmp"filename (one rename clobbers the other's tmp mid-write → corrupt JSON possible), and (b) interleave such that disk ends up with writer B's payload but memory ends up with writer A's (setGeoFiltercalls reorder vs. disk writes). Prior reviewer's "structurally sound, race-test follow-up" stopped at the in-memory race — the disk race is real, just rare. Fix: holdcfgMu(Lock) for the duration ofSaveGeoFilter+setGeoFilter, or introduce a dedicatedsaveMucovering the file I/O. Use a per-process unique tmp suffix (.tmp.<pid>.<nano>) for extra safety.
Prior CHANGES_REQUESTED: all 5 ✅ addressed (cfgMu RWMutex, MaxBytesReader, coord validation, writeEnabled comment, polygon 1000-point cap).
Out-of-scope (non-blocking)
- No
TestGeoFilterConcurrentexercising parallel PUT+GET under-race(prior reviewer already flagged as follow-up). _gfWriteEnabledJS module var is monotonically set to true, never reset on subsequent loads (stale if admin rotates apiKey without page reload)._gfPointspersists across panel reopens — surprising UX after a Cancel.- No
fsyncon tmp file before rename (standard "atomic write" caveat, but no other code in repo does this either).
Each must-fix is small (<15 LoC). Once they land, this is ready.
…ten/meshcore-analyzer into feat/geofilter-m3-customizer
…bot#736) These plan files are workspace-local and must not ship upstream. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ds (Kpa-clawbot#736) Upstream added dbschema to go.mod for both cmd/server and cmd/ingestor but the Dockerfile was not updated. Docker build fails at go mod download because the replace directive resolves to ../../internal/dbschema which is not present in the build context without an explicit COPY. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
All 3 must-fix items are now addressed:
Ready for re-review. |
…ds (Kpa-clawbot#736) Upstream added dbschema to go.mod for both cmd/server and cmd/ingestor but the Dockerfile was not updated. Docker build fails at go mod download because the replace directive resolves to ../../internal/dbschema which is not present in the build context without an explicit COPY. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… GET The GET /api/config/geo-filter endpoint is unauthenticated. Returning writeEnabled (derived from APIKey presence + strength) leaks whether the server has a strong API key configured to anonymous callers. This is a red commit: both subtests now assert writeEnabled is absent. Current production handler still emits the field, so they fail.
PR review found GET /api/config/geo-filter is unauthenticated and was returning writeEnabled (derived from APIKey != "" && !IsWeakAPIKey). That tells any anonymous caller whether a strong API key is configured on the server — low-severity info disclosure, but free to fix. Approach: drop the field entirely instead of gating the endpoint behind requireAPIKey. The polygon itself is intentionally public (read-only clients depend on it), so auth-gating the whole endpoint would break them. Clients that want to write should just attempt PUT and handle 401/403 — the customizer JS already has that error path. - routes.go: remove writeEnabled from both response branches - customize-v2.js: always reveal edit controls; rely on PUT error path - docs/user-guide/geofilter.md: drop the writeEnabled API doc + the "controls only appear when..." UX line Green commit for the red test in the previous commit.
## Summary - Adds `POST /api/admin/prune-geo-filter` endpoint — dry-run by default, `?confirm=true` to permanently delete nodes outside the current geofilter polygon + buffer. Requires `X-API-Key` header. - Adds **Prune nodes** section inside the GeoFilter customizer tab (write-access only, same `writeEnabled` gate as PUT). **Preview** lists affected nodes; **Confirm delete** removes them. - Adds `GetNodesForGeoPrune` and `DeleteNodesByPubkeys` DB helpers. - Updates `docs/user-guide/geofilter.md` — documents the UI button as primary workflow, CLI script as alternative. > **Depends on M3** (`feat/geofilter-m3-customizer`, PR #736). Merge M3 first. ## Test plan - [x] `cd cmd/server && go test ./...` — all pass - [x] Customizer GeoFilter tab without `apiKey` — Prune section not visible - [x] With `apiKey` + polygon active — Prune section visible - [x] **Preview** returns list of nodes outside polygon (no deletions) - [x] **Confirm delete** removes nodes, list clears - [x] `POST /api/admin/prune-geo-filter` without `X-API-Key` → 401 - [x] `POST /api/admin/prune-geo-filter` with no polygon configured → 400 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…ds (Kpa-clawbot#736) Upstream added dbschema to go.mod for both cmd/server and cmd/ingestor but the Dockerfile was not updated. Docker build fails at go mod download because the replace directive resolves to ../../internal/dbschema which is not present in the build context without an explicit COPY. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rebase blocked — needs your inputWe tried to auto-rebase onto current master but hit a 6-file config-design conflict on your first commit:
The bot can't safely guess at the M3 customizer's intended design vs what master has accumulated. Aborting rather than risk breaking your PR's intent. Recommendation: please rebase locally — We'll auto-merge once it's clean. |
|
Confirming the existing CHANGES_REQUESTED review. Two must-fix items: (1) |
Summary
Part of #669 — M3: Customizer integration.
Backend
PUT /api/config/geo-filter(requiresX-API-Key) — savesgeo_filtertoconfig.jsonatomically (temp+rename) and updates in-memory config immediately. No restart needed.SaveGeoFilter()inconfig.goreads config as a raw map (preserving_commentfields and any unknown keys), updates just thegeo_filterkey, writes back.GET /api/config/geo-filternow includeswriteEnabled: bool—truewhen server has a strongapiKeyconfigured.Server.configDirfield wired from-config-dirflag.TestPutConfigGeoFilter× 4,TestSaveGeoFilter× 3, updatedTestConfigGeoFilterEndpoint× 2).Frontend
GET /api/config/geo-filteron tab open; shows current polygon on a Leaflet map.writeEnabled=false). On servers with a write-capableapiKey, the edit UI reveals itself: undo/clear, buffer km, API key input, Save and Remove buttons.PUT /api/config/geo-filter; applies immediately in-memory.Docs
docs/user-guide/geofilter.md: complete guide covering config schema, customizer tab, builder, prune script, and API.docs/user-guide/configuration.mdandcustomization.md.config.example.json_commentto mention the Customizer tab.Test plan
apiKey: GeoFilter tab shows map, no edit controlsapiKey: edit controls appear after tab loadsgeo_filtergo test ./...passes🤖 Generated with Claude Code