Skip to content

Support a builtin audit loader that persists audit events into an internal table #76291

Description

@iatman

Feature request

Is your feature request related to a problem? Please describe.

Analyzing audit data with SQL currently requires installing the external
fe-plugins-auditloader
plugin, which has several long-standing pain points:

  1. Operational cost: users must manually create the audit database/table,
    download a zip, edit plugin.conf, distribute it to every FE, and run
    INSTALL PLUGIN; every StarRocks upgrade requires re-validating plugin
    compatibility.
  2. Security risk: plugin.conf stores a database user and password in
    plaintext (or weakly encrypted) to stream-load audit data back into the
    cluster, which violates common enterprise security baselines.
  3. Version skew: the plugin is released independently from the server and
    uses reflection to stay compatible with multiple releases, so it only
    collects the common subset of audit fields, and new fields lag behind.
  4. Silent mismatch: a plugin/server version mismatch can silently drop
    audit data.

Describe the solution you'd like

Build the audit-loading capability into the FE as a native, toggleable
feature:

  • A builtin AUDIT plugin persists audit events (queries and connections)
    into an internal table _statistics_.starrocks_audit_tbl so audit data
    can be analyzed directly with SQL. No external zip, no plaintext
    credentials: the write path reuses the cluster-internal stream load
    mechanism already used by the query_history system table.
  • Disabled by default and gated by mutable FE configs:
    enable_audit_loader (default false),
    audit_loader_load_interval_seconds (default 60),
    audit_loader_batch_max_bytes (default 50MB, byte-capped per-FE buffer
    that drops and counts events under extreme overload to protect FE
    memory).
  • The audit table is created automatically by the leader FE
    (day-partitioned, partition_live_number=30, recreated automatically if
    dropped by mistake).
  • Conflict protection: while any external dynamic AUDIT plugin (e.g. the
    auditloader zip plugin) is installed, the builtin loader stays inert to
    avoid importing audit data twice, and recovers automatically after the
    external plugin is uninstalled.
  • Reliability rules: batches are only removed from the in-memory buffer
    after a successful load (retry on failure); empty batches are never
    submitted; oversized statements are truncated to the 1MB column width
    instead of failing the whole batch.
  • Coexists with and is independent of the file-based fe.audit.log;
    slow-query classification follows the existing qe_slow_log_ms config.

Describe alternatives you've considered

  1. Keep using the external fe-plugins-auditloader: leaves the plaintext
    credential and version-skew problems unsolved, and requires per-cluster
    manual deployment.
  2. Ship the plugin source inside the main repo but keep the
    INSTALL PLUGIN zip flow: still requires credentials in plugin.conf
    and manual installation on every FE.
  3. Collect fe.audit.log with an external log pipeline (Filebeat/Vector
    etc.) into another system: introduces extra infrastructure and loses
    the ability to analyze audit data in StarRocks with plain SQL.

Additional context

  • Implementation PR: [Feature] Support builtin audit loader to persist audit events into internal table #76287
  • The table schema follows the documented layout of the existing
    audit-loader plugin (see the Audit Loader page in the docs), with the
    stmt column widened to VARCHAR(1048576) and clientIp widened to
    VARCHAR(64) for IPv6 host-port strings.
  • Non-goals for the first version: lossless auditing under extreme
    overload (the buffer is byte-capped by design), a configurable target
    table (the credential-free internal load whitelist requires a fixed
    table), and per-user/expression filtering (may be added later based on
    feedback).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions