You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Analyzing audit data with SQL currently requires installing the external fe-plugins-auditloader
plugin, which has several long-standing pain points:
Operational cost: users must manually create the audit database/table,
download a zip, edit plugin.conf, distribute it to every FE, and run INSTALL PLUGIN; every StarRocks upgrade requires re-validating plugin
compatibility.
Security risk: plugin.conf stores a database user and password in
plaintext (or weakly encrypted) to stream-load audit data back into the
cluster, which violates common enterprise security baselines.
Version skew: the plugin is released independently from the server and
uses reflection to stay compatible with multiple releases, so it only
collects the common subset of audit fields, and new fields lag behind.
Silent mismatch: a plugin/server version mismatch can silently drop
audit data.
Describe the solution you'd like
Build the audit-loading capability into the FE as a native, toggleable
feature:
A builtin AUDIT plugin persists audit events (queries and connections)
into an internal table _statistics_.starrocks_audit_tbl so audit data
can be analyzed directly with SQL. No external zip, no plaintext
credentials: the write path reuses the cluster-internal stream load
mechanism already used by the query_history system table.
Disabled by default and gated by mutable FE configs: enable_audit_loader (default false), audit_loader_load_interval_seconds (default 60), audit_loader_batch_max_bytes (default 50MB, byte-capped per-FE buffer
that drops and counts events under extreme overload to protect FE
memory).
The audit table is created automatically by the leader FE
(day-partitioned, partition_live_number=30, recreated automatically if
dropped by mistake).
Conflict protection: while any external dynamic AUDIT plugin (e.g. the
auditloader zip plugin) is installed, the builtin loader stays inert to
avoid importing audit data twice, and recovers automatically after the
external plugin is uninstalled.
Reliability rules: batches are only removed from the in-memory buffer
after a successful load (retry on failure); empty batches are never
submitted; oversized statements are truncated to the 1MB column width
instead of failing the whole batch.
Coexists with and is independent of the file-based fe.audit.log;
slow-query classification follows the existing qe_slow_log_ms config.
Describe alternatives you've considered
Keep using the external fe-plugins-auditloader: leaves the plaintext
credential and version-skew problems unsolved, and requires per-cluster
manual deployment.
Ship the plugin source inside the main repo but keep the INSTALL PLUGIN zip flow: still requires credentials in plugin.conf
and manual installation on every FE.
Collect fe.audit.log with an external log pipeline (Filebeat/Vector
etc.) into another system: introduces extra infrastructure and loses
the ability to analyze audit data in StarRocks with plain SQL.
The table schema follows the documented layout of the existing
audit-loader plugin (see the Audit Loader page in the docs), with the stmt column widened to VARCHAR(1048576) and clientIp widened to VARCHAR(64) for IPv6 host-port strings.
Non-goals for the first version: lossless auditing under extreme
overload (the buffer is byte-capped by design), a configurable target
table (the credential-free internal load whitelist requires a fixed
table), and per-user/expression filtering (may be added later based on
feedback).
Feature request
Is your feature request related to a problem? Please describe.
Analyzing audit data with SQL currently requires installing the external
fe-plugins-auditloader
plugin, which has several long-standing pain points:
download a zip, edit
plugin.conf, distribute it to every FE, and runINSTALL PLUGIN; every StarRocks upgrade requires re-validating plugincompatibility.
plugin.confstores a database user and password inplaintext (or weakly encrypted) to stream-load audit data back into the
cluster, which violates common enterprise security baselines.
uses reflection to stay compatible with multiple releases, so it only
collects the common subset of audit fields, and new fields lag behind.
audit data.
Describe the solution you'd like
Build the audit-loading capability into the FE as a native, toggleable
feature:
into an internal table
_statistics_.starrocks_audit_tblso audit datacan be analyzed directly with SQL. No external zip, no plaintext
credentials: the write path reuses the cluster-internal stream load
mechanism already used by the
query_historysystem table.enable_audit_loader(defaultfalse),audit_loader_load_interval_seconds(default 60),audit_loader_batch_max_bytes(default 50MB, byte-capped per-FE bufferthat drops and counts events under extreme overload to protect FE
memory).
(day-partitioned,
partition_live_number=30, recreated automatically ifdropped by mistake).
auditloader zip plugin) is installed, the builtin loader stays inert to
avoid importing audit data twice, and recovers automatically after the
external plugin is uninstalled.
after a successful load (retry on failure); empty batches are never
submitted; oversized statements are truncated to the 1MB column width
instead of failing the whole batch.
fe.audit.log;slow-query classification follows the existing
qe_slow_log_msconfig.Describe alternatives you've considered
credential and version-skew problems unsolved, and requires per-cluster
manual deployment.
INSTALL PLUGINzip flow: still requires credentials inplugin.confand manual installation on every FE.
fe.audit.logwith an external log pipeline (Filebeat/Vectoretc.) into another system: introduces extra infrastructure and loses
the ability to analyze audit data in StarRocks with plain SQL.
Additional context
audit-loader plugin (see the Audit Loader page in the docs), with the
stmtcolumn widened toVARCHAR(1048576)andclientIpwidened toVARCHAR(64)for IPv6 host-port strings.overload (the buffer is byte-capped by design), a configurable target
table (the credential-free internal load whitelist requires a fixed
table), and per-user/expression filtering (may be added later based on
feedback).