Skip to content

Add RocksDB action backed by a RocksDB HTTP service - #1266

Merged
Paul Lizer (paullizer) merged 7 commits into
Developmentfrom
paullizer-add-rocksdb-action
Aug 18, 2026
Merged

Paul Lizer (paullizer) merged 7 commits into
Developmentfrom
paullizer-add-rocksdb-action

Conversation

@paullizer

@paullizer Paul Lizer (paullizer) commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a new rocksdb action type so agents can query an ordered RocksDB key-value store, with its own configuration card and Test Connection button in the action modal — similar in shape to the existing SQL and Cosmos actions.

RocksDB is an embedded C++ library with no native network protocol. SimpleChat does not run RocksDB locally and never opens a database directory on the application host. Instead, the action calls a RocksDB-backed HTTP/JSON service that the operator runs alongside their data.

Agent
  └── RocksDbPlugin
        └── requests ──► RocksDB HTTP/JSON service ──► RocksDB database

Agent functions

Reads are always available:

get_value · get_values · key_exists · scan_prefix · scan_range · list_column_families · get_database_stats

Writes (put_value, delete_value, write_batch) are refused unless the action explicitly allows them, matching the read-only default of the SQL action. While read-only they never issue a request.

Configuration

Field Notes
Service Base URL Required; http/https only
Authentication Scheme none, bearer, or api_key (configurable header name)
Verify TLS Certificate Defaults to on
Column Family Defaults to default
Read-Only Defaults to on
Key / Value Encoding utf8, base64, and json for values
Key Prefix Hints Surfaced to the model to focus its scans
Max Results / Max Value Bytes / Timeout Bounded caps

The configured encodings are sent with every service request so the service knows the wire representation of keys and values. The service token lives in auth.key, so existing Key Vault storage and masked-edit flows apply unchanged — no plugin.schema.json change needed.

Security

  • No local filesystem exposure and no path-traversal surface, since SimpleChat never opens a database directory.
  • POST /api/plugins/test-rocksdb-connection sits on the bpap blueprint with @swagger_route(security=get_auth_security())@login_required@user_required.
  • Base URLs restricted to http/https.
  • Service failures report the HTTP status code only — response bodies are never echoed to the browser or the model.
  • Oversized values are truncated with an explicit value_truncated flag plus the original value_bytes, so one record cannot flood the model context.

Service contract

The plugin speaks a documented SimpleChat contract: /health, /get, /multi_get, /exists, /scan, /column_families, /stats, /put, /delete, /batch. Range semantics (inclusive start, exclusive end, reverse, clamped limit) are specified in the feature doc so operators can implement a conforming service. The Test Connection button probes GET /health and falls back to POST /scan with {"limit": 1} when /health returns 404.

Validation

Suite Result
functional_tests/test_rocksdb_plugin.py 10/10
functional_tests/route_tests/ 12/12
functional_tests/test_cosmos_query_plugin.py (regression) 4/4
functional_tests/test_yamcs_action_plugin.py (post-merge regression) 14/14
functional_tests/test_agent_modal_instructions_step_order.py (post-merge regression) 6/6
ui_tests/test_workspace_rocksdb_action_modal.py 3 collected, env-gated skip per repo convention
Playwright DOM pass over the action modal 0 console errors

I drove the real modal in a headless browser rather than only parsing the JS. That caught two genuine bugs, both fixed here:

  1. The summary card displayed the un-normalized base URL (trailing slash) while the saved endpoint stripped it.
  2. "bad block" was misclassified as a lock error because block contains lock.

After merging Development I re-ran the DOM pass to confirm the RocksDB and Yamcs sections both still switch, configure, and post correctly.

Files

New

  • application/single_app/semantic_kernel_plugins/rocksdb_plugin.py
  • application/single_app/static/json/schemas/rocksdb.definition.json
  • functional_tests/test_rocksdb_plugin.py
  • ui_tests/test_workspace_rocksdb_action_modal.py
  • docs/explanation/features/v0.250.215/ROCKSDB_ACTION.md

Modified

  • route_backend_plugins.py — test-connection route and action type card
  • semantic_kernel_plugins/plugin_health_checker.py — manifest validation
  • templates/_plugin_modal.html — config section, summary card, CSS with dark theme
  • static/js/plugin_modal_stepper.js — full type wiring
  • static/js/workspace/view-utils.js — action type icon
  • config.py (VERSION0.250.215), release notes, features index, logging tags

Review notes

Commit history. The first commit added the action with both an embedded mode (via the rocksdict binding, gated by a ROCKSDB_ALLOWED_ROOTS path allowlist) and the remote mode. The second commit removes embedded support entirely per the decision not to run RocksDB on the app host — dropping the rocksdict dependency, the allowlist env var, the embedded manifest fields, and the connection-mode selector. Two later commits merge Development. Reviewing the squashed diff is by far the clearest view.

Merge conflicts. Development shipped the Yamcs action, which touches the same action-type integration points, so the automatic merge interleaved the two features. They were resolved to keep both side by side: separate routes, separate method blocks, and merged isStructuredConfigType / summary-type conditions covering isRocksDbType and isYamcsType. The net diff against Development is +2852/−6, and every one of those 6 deleted lines is either the version bump, a trailing-whitespace line, or a condition that gains isRocksDbType while keeping isYamcsType. No Development code was lost.

Known limitations

  • The action depends on a RocksDB HTTP service that you operate; SimpleChat does not ship one.
  • The contract is SimpleChat-defined, so an existing RocksDB proxy will usually need a thin adapter.
  • Results are capped per call, so very large scans must be paged by the agent using narrower prefixes or range bounds.

Follow-up

No GitHub issue is currently linked — happy to file one if you would like this tracked.

Paul Lizer (paullizer) and others added 2 commits August 17, 2026 21:09
Adds a new `rocksdb` action type so agents can query an ordered RocksDB
key-value store, with its own configuration card and Test Connection
button in the action modal.

RocksDB is an embedded library rather than a network database server, so
the action supports two connection modes:

- Embedded: opens a RocksDB directory on the application host through the
  `rocksdict` binding, using read-only or secondary access.
- Remote: calls a RocksDB-backed HTTP/JSON service with optional bearer or
  API-key authentication.

Reads are always available (get_value, get_values, key_exists, scan_prefix,
scan_range, list_column_families, get_database_stats). Writes (put_value,
delete_value, write_batch) are refused unless the action explicitly allows
them, matching the read-only default of the SQL action.

Security:

- Embedded paths are gated by a new ROCKSDB_ALLOWED_ROOTS environment
  variable and validated with realpath plus a common-prefix check, so
  traversal, symlink escapes, and name-prefix siblings are rejected.
  Embedded mode stays disabled while the variable is unset.
- The test-connection route requires an authenticated user and returns
  sanitized errors that never expose filesystem layout or driver text.
- Remote base URLs are restricted to http and https, and service failures
  report the HTTP status code only.

The `rocksdict` wheel is imported lazily, so an image without it still
starts, still lists the action type, and still supports remote mode.

Validation: 8/8 RocksDB functional tests including a real embedded RocksDB
round trip, 12/12 route policy tests, 4/4 Cosmos regression tests, and a
Playwright DOM pass over the action modal with no console errors.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
SimpleChat will not run RocksDB locally, so the action no longer opens a
RocksDB database directory on the application host. The `rocksdb` action
now talks exclusively to a RocksDB-backed HTTP/JSON service.

Removed:

- The `rocksdict` dependency and its lazy import helper.
- The `ROCKSDB_ALLOWED_ROOTS` environment variable and the path allowlist
  resolution it guarded, along with the embedded connection test branch and
  its filesystem error sanitizer.
- Embedded manifest fields (`connection_mode`, `db_path`, `access_mode`,
  `secondary_path`) plus their validation, modal controls, and summary rows.
- Embedded scan, batch, and key/value codec helpers along with the cached
  RocksDB handle map.

Because the connection mode selector is gone, the modal now presents the
service configuration directly, which removes a step from the flow.

The configured `key_encoding` and `value_encoding` are now sent with every
service request so the service knows the wire representation of keys and
values, and the documented contract records that.

Read and write surfaces are unchanged: the same ten kernel functions are
exposed and writes stay blocked until an action explicitly allows them.

Net effect is -1658/+768 lines across 13 files.

Validation: 10/10 RocksDB functional tests, 12/12 route policy tests, 4/4
Cosmos regression tests, and a Playwright DOM pass over the simplified
action modal with no console errors, including assertions that the embedded
controls are absent from the DOM.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer Paul Lizer (paullizer) changed the title Add RocksDB action with embedded and remote connection modes Add RocksDB action backed by a RocksDB HTTP service Aug 18, 2026
Paul Lizer (paullizer) and others added 2 commits August 17, 2026 22:45
Development added the Yamcs action, which touches the same action-type
integration points as RocksDB, so the two features were interleaved by the
automatic merge. Resolved to keep both side by side.

Conflicts resolved:

- config.py: bumped VERSION to 0.250.214, above Development's 0.250.213.
- route_backend_plugins.py: kept both test_yamcs_connection and
  test_rocksdb_connection as separate routes.
- plugin_modal_stepper.js: kept both test-connection button listeners, both
  method blocks, and merged isStructuredConfigType and the summary
  database-type conditions to cover isRocksDbType and isYamcsType.
- release_notes.md: kept both sets of notes and renumbered the RocksDB
  entry to v0.250.214 so it sits above Development's v0.250.213.

The RocksDB feature doc moved to docs/explanation/features/v0.250.214/ and
its version references, the features index link, and the functional and UI
test version headers were updated to match.

Validation after the merge: 10/10 RocksDB functional tests, 14/14 Yamcs
functional tests, 4/4 Cosmos regression tests, 12/12 route policy tests,
and a Playwright DOM pass confirming both the RocksDB and Yamcs sections
switch, configure, and post correctly with no console errors.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Development shipped the agent instruction context references feature and
claimed VERSION 0.250.214, which collided with the RocksDB entry.

Conflict resolved:

- release_notes.md: split the shared v0.250.214 heading so Development's
  agent instruction feature keeps v0.250.214 and RocksDB moves to its own
  v0.250.215 section above it.

Renumbered alongside it: config.py VERSION, the feature doc folder
(docs/explanation/features/v0.250.215/), the features index link, the
plugin module docstring reference, and the functional and UI test headers.

Validation: 10/10 RocksDB functional tests, 14/14 Yamcs functional tests,
6/6 agent modal instruction step-order tests, 4/4 Cosmos regression tests,
and 12/12 route policy tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread application/single_app/semantic_kernel_plugins/rocksdb_plugin.py Fixed
Comment thread application/single_app/route_backend_plugins.py Fixed
Comment thread application/single_app/route_backend_plugins.py Fixed
Comment thread application/single_app/route_backend_plugins.py Fixed
Comment thread application/single_app/route_backend_plugins.py Fixed
Comment thread application/single_app/route_backend_plugins.py Fixed
Comment thread functional_tests/test_rocksdb_plugin.py Fixed
Comment thread functional_tests/test_rocksdb_plugin.py Fixed
Paul Lizer (paullizer) and others added 3 commits August 17, 2026 23:07
CodeQL flagged 15 new alerts on this PR. Fixed the ones this change
introduced.

Always verify TLS (py/request-without-cert-validation, high):
Removed the "Verify TLS Certificate" toggle and the verify_tls field
instead of passing a possibly-false verify= into requests. Certificate
validation is now always enforced. Operators fronting the service with a
private certificate authority should install that CA in the application
trust store or point REQUESTS_CA_BUNDLE at it, which is the documented
path. This drops a config field, a UI control, and a request parameter.

Keep exception text out of logs (py/clear-text-logging-sensitive-data,
high): the plugin and the connection-test route no longer interpolate raw
exceptions into log messages. They log the exception type plus a sanitized
detail through sanitize_log_message, so a transport error carrying request
details cannot reach the log sink verbatim.

Stop echoing exception text to the browser (py/stack-trace-exposure,
medium): the connection-test route returns authored messages for invalid
base URLs, action lookup failures, and unresolved stored secrets rather
than str(exc).

Close test file handles (py/file-not-closed): the functional test reads the
plugin module and requirements.txt through context managers.

The remaining alerts on this PR are pre-existing on Development and are not
introduced here: the functions_appinsights.py clear-text-logging and
log-injection sinks, py/import-of-mutable-attribute in scripts/ and
deployers/, and the same py/stack-trace-exposure pattern already open on
the Yamcs and Cosmos routes.

Validation: 10/10 RocksDB functional tests, 14/14 Yamcs functional tests,
12/12 route policy tests, and a Playwright DOM pass confirming the modal
still configures and posts correctly with the TLS control removed and no
console errors.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
CodeQL still traced a path from the RocksDB action into the shared logging
sinks in functions_appinsights.py, so the action no longer puts anything
credential-derived into log properties at all.

- The initialization log dropped has_auth_key. It read the credential to
  compute a boolean, and auth_scheme already says whether the action
  authenticates.
- The plugin and connection-test failure logs now record only the exception
  type. Both already pass exceptionTraceback=True, so Application Insights
  still captures the exception itself, and the message text added nothing
  that was not already available.

Errors returned to the model still go through sanitize_log_message, which
redacts secret assignments and Authorization values.

Validation: 10/10 RocksDB functional tests and 12/12 route policy tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Development shipped the retrieved-sources and cited-references fix and
claimed VERSION 0.250.215, which collided with the RocksDB entry.

Conflict resolved:

- release_notes.md: split the shared v0.250.215 heading so Development's
  citation fix keeps v0.250.215 and RocksDB moves to its own v0.250.216
  section above it.

Renumbered alongside it: config.py VERSION, the feature doc folder
(docs/explanation/features/v0.250.216/), the features index link, the
plugin module docstring reference, and the functional and UI test headers.

Validation: 10/10 RocksDB functional tests, 16/16 cited-source tracking
tests, 14/14 Yamcs functional tests, and 12/12 route policy tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) merged commit de3fd4a into Development Aug 18, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants