feat(sql): add native query support for sys.tasks

[FrankChen021]

Depends on apache/druid#20183, which provides the native system-table execution framework and sys.server_properties support. This stacked PR contains only the sys.tasks integration.

Description

This PR adds native-query support for sys.tasks on top of the framework introduced by apache#20183. It registers the task table descriptor and Overlord data provider, exposes the Calcite native table representation, and pushes eligible task filters into metadata storage while retaining the original native filter on the Broker for final correctness.

The implementation keeps storage capabilities explicit:

• MetadataTaskStorage accepts validated filters through TaskStorageQueryFilter.
• Metadata SQL generation remains in the metadata action handler.
• Equality and IN predicates use database-specific SqlDialect capability checks.
• Status filters prune active or completed task lookups without pretending that status is a physical metadata column.
• Storage implementations that cannot safely push filters down retain the existing behavior.

Benchmark

The added JMH benchmark executes complete SQL statements through Druid's SQL statement and native-query layers against 100,000 migrated task records in embedded Derby. Each operation includes SQL planning, metadata retrieval, residual-filter evaluation, native scan processing, Broker-side expressions/aggregation/sorting, and final result materialization. The Bindable and native paths use the same Derby task table and in-process Overlord/node adapters; node discovery and HTTP transport are excluded from both paths.

The full-list workload is the exact task-list SQL issued by the Web Console, including its computed status and non-time ordering. Three GROUP BY workloads cover the Web Console status summary, unfiltered datasource grouping, and grouping after a pushed datasource filter.

Short local run on JDK 25: one fork, two 1-second warmup iterations, and five 1-second measurement iterations.

SQL workload	Bindable	Native	Relative result
Web Console full task list	357.919 ms/op	522.178 ms/op	0.69x (native 46% slower)
task_id = 'task-050000'	310.146 ms/op	1.428 ms/op	217x faster
datasource = 'datasource-000'	345.465 ms/op	5.789 ms/op	60x faster
datasource = 'datasource-000' AND type = 'type-0'	308.093 ms/op	3.093 ms/op	100x faster
group_id = 'group-0000'	327.256 ms/op	90.285 ms/op	3.6x faster
GROUP BY effective task status	325.520 ms/op	362.293 ms/op	0.90x (native 11% slower)
GROUP BY datasource	325.269 ms/op	387.068 ms/op	0.84x (native 19% slower)
WHERE datasource = 'datasource-000' GROUP BY type	305.367 ms/op	7.940 ms/op	38x faster

Both datasource and type are pushed into the metadata predicate for the combined-filter case. task_id, datasource, type, and group_id equality predicates are all eligible for pushdown. Derby has an existing datasource/active index, while type and group_id are not indexed; this explains why task ID and datasource filters show the largest gains and the group ID scan improves more modestly.

The unfiltered workloads show the current limitation honestly: components return scan rows, while aggregation and global ordering remain on the Broker, so there is no metadata-pruning benefit and the native machinery adds overhead. These numbers are directional in-process comparisons, not production latency claims.

Validation

• TasksTableDataProviderTest, SqlDialectTest, and SQLMetadataStorageActionHandlerTest: 62 tests passed.
• NativeSysTasksQueryTest: 20 embedded end-to-end tests passed, including both COUPLED and DECOUPLED planning and the Web Console query.
• A focused CalciteQueryTest passed with the SQL test framework's data-source query-handler integration.
• Benchmark module compilation and assembly passed.
• The SQL benchmark completed all eight workloads for both Bindable and native paths.

Release note

SQL queries against sys.tasks can opt into the native query engine with useNativeQueryForSystemTables=true. Supported filters are pushed into metadata task storage, while unsupported system tables and storage implementations retain their existing execution behavior.

---

Key changed/added classes in this PR

• TasksTableDataProvider
• TaskTableDescriptor
• NativeTasksTable
• TaskStorageQueryFilter
• SqlDialect
• SqlPredicateBuilder
• SysTasksTableBenchmark

---

This PR has:

• been self-reviewed.
• added documentation for the new behavior.
• included a release note in the PR description.
• added comments and Javadocs where the intent is not obvious.
• added unit and embedded end-to-end tests.
• included a reproducible SQL benchmark with 100,000 task records.

[Copilot]

🟡 Changes recommended

TaskTableDescriptor row authorization does not handle null datasource values and can fail sys.tasks queries (and/or make unsafe authorization decisions) when task metadata lacks a datasource.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds native-query execution support for sys.tasks by registering a system-table descriptor and Overlord-backed data provider, exposing a Calcite native table representation, and enabling safe metadata filter pushdown (with residual filtering preserved for correctness).

Changes:

• Register sys.tasks as a native system table (descriptor + Overlord data provider) and expose a native Calcite table (NativeTasksTable).
• Add task-filter pushdown plumbing into metadata storage (validated filter wrapper, SQL predicate generation, dialect capability gating).
• Add/adjust unit, embedded end-to-end, and JMH benchmark coverage for the native sys.tasks path.

File summaries

File	Description
sql/src/test/java/org/apache/druid/sql/calcite/schema/SystemTableDataProviderTest.java	Extends native-table metadata assertions to include sys.tasks and updates native system table selection mocks.
sql/src/test/java/org/apache/druid/sql/calcite/schema/SystemSchemaTest.java	Aligns sys.tasks type verification with TaskTableDescriptor signature.
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchemaProvider.java	Ensures sys.tasks table name is sourced from the new descriptor constant.
sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java	Wires sys.tasks to use TaskTableDescriptor signature and provides a native representation via NativeSystemTable.
sql/src/main/java/org/apache/druid/sql/calcite/schema/NativeTasksTable.java	Adds native Calcite table implementation for sys.tasks.
services/src/main/java/org/apache/druid/cli/CliOverlord.java	Registers the Overlord sys.tasks data provider binding.
server/src/test/java/org/apache/druid/server/system/handler/SystemTableQueryClientTest.java	Uses TaskTableDescriptor to exercise leader-only routing behavior for sys.tasks.
server/src/main/java/org/apache/druid/server/system/table/TaskTableDescriptor.java	Introduces native descriptor for sys.tasks including routing mode, row signature, and row authorization.
server/src/main/java/org/apache/druid/server/system/module/SystemTableModule.java	Registers the sys.tasks routing descriptor alongside existing native system tables.
indexing-service/src/test/java/org/apache/druid/metadata/SQLMetadataStorageActionHandlerTest.java	Adds coverage for metadata-side task status listing with validated pushdown filters and SQL generation variants.
indexing-service/src/test/java/org/apache/druid/metadata/SqlDialectTest.java	Adds tests validating dialect capability decisions for equality/IN across sensitive string values.
indexing-service/src/test/java/org/apache/druid/indexing/overlord/task/TasksTableDataProviderTest.java	Adds unit tests for pushdown filter extraction and Overlord-provider behavior under leadership transitions.
indexing-service/src/main/java/org/apache/druid/metadata/TaskStorageQueryFilter.java	Adds immutable wrapper for validated filters, including status-based pruning of active/complete lookups.
indexing-service/src/main/java/org/apache/druid/metadata/SqlPredicateBuilder.java	Adds SQL predicate + binding generation for validated task filters, keeping SQL/bindings in sync.
indexing-service/src/main/java/org/apache/druid/metadata/SQLMetadataStorageActionHandler.java	Integrates filter-aware task-status list retrieval and pushes eligible predicates into metadata SQL.
indexing-service/src/main/java/org/apache/druid/metadata/SqlDialect.java	Adds dialect capability model for deciding which validated filters are safe to translate to SQL.
indexing-service/src/main/java/org/apache/druid/metadata/MetadataStorageActionHandler.java	Adds a default filter-aware task-status-list method for storage handlers.
indexing-service/src/main/java/org/apache/druid/metadata/DerbyMetadataStorageActionHandler.java	Declares Derby dialect capabilities for task-filter SQL generation.
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskStorage.java	Adds filter-support capability flag and filter-aware status retrieval default method.
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueryTool.java	Adds filtered-status query entry point, leader-safety checks, and routes to storage’s filter-aware methods when present.
indexing-service/src/main/java/org/apache/druid/indexing/overlord/task/TasksTableDataProvider.java	Implements the Overlord native row supplier for sys.tasks, including pushdown filter advertisement.
indexing-service/src/main/java/org/apache/druid/indexing/overlord/MetadataTaskStorage.java	Implements filter-support and routes filtered lookups into the metadata handler.
extensions-core/postgresql-metadata-storage/src/main/java/org/apache/druid/metadata/PostgreSQLMetadataStorageActionHandler.java	Declares Postgres dialect capabilities for task-filter SQL generation.
extensions-core/mysql-metadata-storage/src/main/java/org/apache/druid/metadata/MySQLMetadataStorageActionHandler.java	Declares MySQL dialect capabilities for task-filter SQL generation.
extensions-contrib/sqlserver-metadata-storage/src/main/java/org/apache/druid/metadata/SQLServerMetadataStorageActionHandler.java	Declares SQL Server dialect capabilities for task-filter SQL generation.
embedded-tests/src/test/java/org/apache/druid/testing/embedded/query/NativeSysTasksQueryTest.java	Adds embedded end-to-end coverage for native sys.tasks planning and residual filter behavior.
docs/querying/sql-metadata-tables.md	Updates docs examples and adds sys.tasks to the native system-table list and behavior notes.
benchmarks/src/test/java/org/apache/druid/metadata/SysTasksTableBenchmark.java	Adds JMH benchmark isolating metadata-read improvements from pushdown vs bindable paths.

Review details

• Files reviewed: 28/28 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.