Skip to content

feat(scan): add application data-boundary diagnostic mode#621

Open
fraware wants to merge 14 commits into
open-jarvis:mainfrom
fraware:main
Open

feat(scan): add application data-boundary diagnostic mode#621
fraware wants to merge 14 commits into
open-jarvis:mainfrom
fraware:main

Conversation

@fraware

@fraware fraware commented Jul 1, 2026

Copy link
Copy Markdown

Summary

This PR extends jarvis scan with a new application data-boundary mode:

jarvis scan --data-boundaries
jarvis scan --data-boundaries --json
jarvis scan --data-boundaries --strict

The new mode reports configured OpenJarvis data surfaces such as cloud-capable
inference settings, memory-to-cloud composition, traces, learning/training,
web search, MCP, browser/local tools, server/A2A exposure, messaging channels,
connector credential files, local runtime stores, and API-key environment
variables.

The scanner inspects configuration values, environment-variable presence, and file existence.
It does not read connector-token contents, memory databases, trace databases, prompts,
logs, OAuth scopes, browser localStorage, Tauri storage, or provider policies.

Why

OpenJarvis is local-first while also supporting optional cloud engines,
connectors, tools, channels, traces, memory, learning, and optimization. Users
need a simple way to verify whether a configuration is local-only, cloud-capable,
or a mixed setup where local memory may be injected into cloud-bound prompts.

The existing jarvis scan command audits host/environment posture. This PR adds
an application-level counterpart without creating a competing top-level command.

What changed

  • Added openjarvis.security.data_boundary_audit with structured findings and
    redacted JSON output.
  • Added jarvis scan --data-boundaries with table and JSON output.
  • Added --show-paths for local debugging while keeping paths redacted by
    default.
  • Added --strict to fail on either fail or warn findings.
  • Honored OPENJARVIS_CONFIG so the scan audits the same explicit config file
    used by the runtime.
  • Reported OpenJarvis home/config-root resolution errors as findings rather than
    crashing.
  • Skipped the global update check only for jarvis scan --data-boundaries, so
    this diagnostic mode does not trigger the normal update-check thread.
  • Separated channel secrets from channel endpoints/identifiers and added local
    messaging credential-directory checks.
  • Added local-store permission warnings for symlinked or group/other-readable
    stores on POSIX systems.
  • Added tests for core report construction, redaction, cloud/memory composition,
    tools, MCP, channels, env credentials, strict mode, config/root-load failures, and
    normal jarvis scan regression coverage.
  • Added user documentation.

Non-goals

Test plan

  • uv run ruff check on the five data-boundary files (via ruff check after uv sync failed with TLS UnknownIssuer)
  • pytest tests/security/test_data_boundary_audit.py tests/cli/test_scan_data_boundaries.py -v (54 passed, 2 skipped on clean re-run)
  • python -m compileall on data_boundary_audit.py, scan_cmd.py, cli/__init__.py
  • jarvis scan --json (exit 0)
  • jarvis scan --quick --json (exit 0)
  • jarvis scan --data-boundaries (exit 0 with UTF-8 no-BOM config.toml)
  • jarvis scan --data-boundaries --json (exit 0)
  • jarvis scan --data-boundaries --json --show-paths (exit 0)
  • jarvis scan --data-boundaries --strict exits 1 when fail findings present

Local verification

Because the local Windows environment had a PyPI TLS UnknownIssuer issue with uv sync, I verified using an editable install plus trusted-host pip fallback. The CI workflows are currently waiting for maintainer approval on this fork PR.

Step Result
Targeted Ruff check on changed Python files PASS
Targeted pytest suite PASS — 54 passed, 2 skipped
python -m compileall on changed source modules PASS
jarvis scan --json PASS
jarvis scan --quick --json PASS
jarvis scan --data-boundaries PASS
jarvis scan --data-boundaries --json PASS
jarvis scan --data-boundaries --json --show-paths PASS
jarvis scan --data-boundaries --strict PASS — exits 1 when fail findings are present
OPENJARVIS_CONFIG custom TOML smoke test PASS — config_loaded: true
Secret-redaction smoke test PASS — env var values and connector JSON contents are not printed

Follow-up validation

  • Fixed Ruff top-level blank-line spacing in tests/cli/test_scan_data_boundaries.py.
  • Re-ran targeted ruff check on changed Python files.
  • Re-ran targeted pytest suite.
  • Re-ran compileall on changed source modules.
  • Hardened _low_noise_config() in both data-boundary test modules against JarvisConfig default store paths and optional warn sources (37a4f692 on origin/main); targeted pytest: 54 passed, 2 skipped.

fraware and others added 11 commits July 1, 2026 01:28
feat(scan): add application data-boundary diagnostic mode
Drop the one-off apply patch helper, dedupe security user-guide copy, and add a CLI integration test ensuring scan --data-boundaries skips update checks.
Add integration test for scan --data-boundaries update-check suppression and replace duplicated three-layer section with a short link.
chore: data-boundary scan PR cleanup
feat(scan): add application data-boundary diagnostic mode
@fraware

fraware commented Jul 1, 2026

Copy link
Copy Markdown
Author

Pushed a small test-formatting cleanup and re-ran the targeted local checks.

@fraware

fraware commented Jul 1, 2026

Copy link
Copy Markdown
Author

Follow-up: hardened _low_noise_config()\ in both data-boundary test modules so JarvisConfig defaults (absolute store paths under the real OPENJARVIS_HOME, MCP enabled, etc.) no longer leak warn findings into info-only scenarios. Targeted suite re-run: 54 passed, 2 skipped. Pushed to \origin/main\ as \37a4f692.

@fraware

fraware commented Jul 1, 2026

Copy link
Copy Markdown
Author

The latest push includes the test cleanup and the _low_noise_config()\ hardening noted above. The targeted local suite is green: 54 passed, 2 skipped.

@fraware

fraware commented Jul 1, 2026

Copy link
Copy Markdown
Author

Follow-up: preserved configured store paths for POSIX symlink detection while keeping resolved-path deduplication. Linux-targeted local-store permission tests pass, including test_symlink_local_store_permission_warns (POSIX symlink/group-permission cases skip locally on Windows; fix validated via full security/CLI pytest suite here). Targeted Ruff, pytest, and compileall are green.

@ElliotSlusky

Copy link
Copy Markdown
Collaborator

Hey @fraware, thank you for the PR! I have a few fixes for you:

  1. Blocking: formatter check fails
    ruff format --check would reformat:
    scan_cmd.py, data_boundary_audit.py, test_scan_data_boundaries.py.
    The repo checklist includes format checking, so this should be fixed before merge.

  2. High: redacted output can leak channel.default_channel
    data_boundary_audit.py appends the raw default_channel value into evidence. That value can contain user identifiers like phone-number-style or chat-channel targets. For a “paste-safe” audit report, this should be presence-only, e.g. “default_channel is set; value redacted”.

  3. High: WhatsApp Baileys auth directory check looks at the wrong default path
    data_boundary_audit.py checks whatsapp_auth, but the actual Baileys default is whatsapp_baileys_bridge/auth from whatsapp_baileys.py and whatsapp_baileys.py.
    This creates a false negative for the real default credential directory.

  4. High: tool detection misses real registered tool names
    data_boundary_audit.py only checks names like browser_open, but the app registers tools like browser_navigate, browser_click, browser_type, browser_screenshot, and browser_extract.
    Same problem at data_boundary_audit.py: local-access detection misses real sensitive tools like file_write, apply_patch, docker_shell_exec, db_query, knowledge_sql, memory_manage, and repl. This weakens the audit’s main purpose.

  5. Medium: cloud key scan misses GEMINI_API_KEY
    data_boundary_audit.py includes GOOGLE_API_KEY but not GEMINI_API_KEY, even though the runtime supports both. That is another false negative.

Verification

Passed:

OPENJARVIS_HOME=/tmp/openjarvis-test-home-621 PYTHONPATH=src python3 -m pytest tests/security/test_data_boundary_audit.py tests/cli/test_scan_data_boundaries.py -q
# 56 passed

python3 -m ruff check src/openjarvis/security/data_boundary_audit.py src/openjarvis/cli/scan_cmd.py src/openjarvis/cli/__init__.py tests/security/test_data_boundary_audit.py tests/cli/test_scan_data_boundaries.py

python3 -m compileall -q src/openjarvis/security/data_boundary_audit.py src/openjarvis/cli/scan_cmd.py src/openjarvis/cli/__init__.py

Failed:

python3 -m ruff format --check ...
# 3 files would be reformatted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants