Compaction failure and storage bloat caused by large JSON field in shared-data PRIMARY KEY table

[0akarma]

Steps to reproduce the behavior (Required)

1. Create a shared-data / cloud-native PRIMARY KEY table with large JSON columns, for example:

   CREATE TABLE history (
     id VARCHAR(32) NOT NULL,
     `@dt` DATETIME NOT NULL,
     ts INT(11) NOT NULL,
     doc JSON NOT NULL
   )
   PRIMARY KEY(id)
   PARTITION BY RANGE(`@dt`) (...)
   DISTRIBUTED BY HASH(id) BUCKETS ...
   PROPERTIES ("replication_num" = "1");

2. Insert records where the doc column may contain very large JSON values (single row can be > 16MB).
3. Keep writing / running compaction on the table.
4. Observe FE log during compaction and autovacuum.

Expected behavior (Required)

• Compaction should complete normally for the table/partitions.
• Autovacuum should be able to reclaim obsolete files.
• Object storage usage should gradually decrease after data becomes inactive.

Real behavior (Required)

• Compaction repeatedly fails for some partitions with: Compaction job txnId=... failed: string length(...) > limit(1048576)
• AutovacuumDaemon keeps running, but vacuumedFiles=0 / vacuumedFileSize=0 for affected partitions.
• The table's storage path keeps accumulating many small objects / historical files.
• Object storage usage stays much larger than the visible table size.

Additional context

• The table is a shared-data / cloud-native PRIMARY KEY table.
• The table contains large JSON columns, especially doc.
• SHOW TABLET reports MinVersion = 0 for tablets, so the issue does not look like tablet version retention.
• This looks like compaction cannot handle oversized stringified metadata / payload for some partitions, which prevents file merging and cleanup.

StarRocks version (Required)

• You can get the StarRocks version by executing SQL select current_version()

current_version()|
-----------------+
4.0.6-45fdede    |

[0akarma]

Additional evidence from our cluster:

This does not appear to originate from FE scheduling or autovacuum itself.
The first failure happens on CN/BE during lake compaction, and FE only reports the same failure at the partition/job level.

BE/CN logs show compaction failure on affected tablets with errors like:

Invalid argument: string length(12xxxxxx) > limit(1048576), string: {
"large_json_array": [
{
"field_a": "...",
"field_b": "...",
...
}
]
}

So the string printed in the error is actual business JSON payload content from our table, not just a small internal identifier or profile string.

FE logs repeatedly report the same compaction failure for the same affected partitions with different txn IDs, which suggests the partitions repeatedly re-enter compaction and repeatedly fail on the same data pattern.

Table characteristics:

• shared-data / cloud-native table
• PRIMARY KEY table
• multiple JSON/text-like columns
• some single-row JSON documents may exceed 1 MB and can be much larger

Impact:

• compaction cannot complete for affected tablets/partitions
• old files are not reclaimed as expected
• object storage usage keeps growing far beyond logical table size

In our case:

• logical table size is much smaller than actual object storage usage
• bucket versioning is NOT enabled
• the storage bloat correlates with repeated compaction failures on the same partitions

From the current evidence, the key problem seems to be that the compaction path hits an internal 1 MB string limit when processing large JSON/text content, which then blocks file reclamation for affected shared-data PK tablets.

If needed, we can provide:

• a redacted schema
• a redacted sample row pattern
• more redacted CN/BE and FE log snippets around the failing compaction tasks