unique index across time partition performance test for updates#6024
unique index across time partition performance test for updates#6024dorinhogea wants to merge 2 commits into
Conversation
dorinhogea
commented
Jun 13, 2026
dorinhogea
commented
Time partitions spread data across multiple shard tables to implement data retention. Until now, unique indexes were only enforced within the shard being written, making it possible to insert the same key into different shards without conflict. A new 'partition_unique' tunable enables cross-shard unique enforcement for TRUNCATE-rollout partitions. When on, any write that would violate a unique constraint in any sibling shard is rejected with the same error that a within-shard violation produces. The enforcement covers inserts and updates that change a key column. The feature carries a measurable write cost proportional to the number of sibling shards and unique indexes, driven by the extra index lookups per write and the larger BDB lock footprint per transaction. A new 'partition_unique_debug' tunable traces the full lifecycle: OSQL_PARTITION_SHARDS send on the replicant, receive/store and free on the master, and each cross-shard index probe. Known limitations: - Enabling the tunable does not validate pre-existing data. - UPSERT is not supported on time partitions when enabled (enforced at write time with a clear error). - ON UPDATE CASCADE is handled master-side in constraints.c without going through the OSQL stream, so cross-shard enforcement does not apply to cascaded key changes until that path is updated. Bug fixes applied during review: - Endian-safe nshards in OSQL_PARTITION_SHARDS wire format (htonl/ntohl) - Bounds-check nshards and validate shard name NUL terminators in receiver - Fail writes on malloc failure in timepart_get_shard_names via errstat - Remove redundant forward declaration of check_cross_shard_unique Refactoring: - Consolidate 7 copy-pasted reqerrstr blocks into reqerrstr_dup_key / reqerrstr_uncommittable_dup helpers in indices.c Tests: add partition_unique_check correctness test and partition_unique_perf performance test. Signed-off-by: Dorin Hogea <dhogea@bloomberg.net>
dorinhogea
force-pushed
the
partuniqindexperf
branch
from
6aa98ca to
8bde8f3
Compare
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: 11/681 tests failed ⚠.
The first 10 failing tests are:
sc_lotsoftables_logicalsc_generated [setup failed]
queuedb_rollover [setup failed] **quarantined**
cinsert_linearizable [failed with core dumped]
systable_locking [failed with core dumped]
socksql_master_swings [failed with core dumped]
random_osql_replay [failed with core dumped]
cldeadlock [failed with core dumped]
sc_resume [db unavailable at finish]
selectv_deadlock
deadlock_load_noreorder_generated
Benchmark measures the overhead of cross-shard unique index enforcement on TRUNCATE time partitions. Runs UPDATE workloads under three modes (off, on, on-nolocks) and reports timing, deadlocks, and lock stats. Parameters: ROWS_PER_SHARD (rows per shard, default 10000), NRANGES (row ranges / max workers, default 8), NWORKERS (parallel writers, default NRANGES), ITERATIONS (benchmark repetitions, default 3), TXN_SIZE (rows per transaction, default 10), D_SIZE (filler column size in bytes, default 16), DEBUG_TRACE (enable detailed lock counters, default 1). Signed-off-by: Dorin Hogea <dhogea@bloomberg.net>
dorinhogea
force-pushed
the
partuniqindexperf
branch
from
8bde8f3 to
0c5c6b1
Compare
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: 10/681 tests failed ⚠.
The first 10 failing tests are:
sc_lotsoftables_logicalsc_generated [setup failed]
queuedb_rollover_noroll1_generated [setup failed] **quarantined**
cinsert_linearizable [failed with core dumped]
random_osql_replay [failed with core dumped]
cldeadlock [failed with core dumped]
sc_resume_logicalsc_generated [db unavailable at finish] **quarantined**
sc_timepart [db unavailable at finish] **quarantined**
selectv_deadlock
deadlock_load_noreorder_generated
deadlock_load
2 participants
