JMT data is very large and should be optimized..

[stringhandler]

Bounty: JMT Data Optimization

Tier: L — 150,000 XTM

Description

Jellyfish Merkle Tree (JMT) data is the single largest consumer of storage in the base node database — jmt_node_data alone accounts for 6.4 GB out of 11.4 GB total. Investigate the root cause of the excessive size (9.6M entries averaging 669 bytes each), propose an optimization strategy, and implement it. This likely involves changes to node serialization, pruning of historical JMT nodes, or restructuring how JMT data is stored in LMDB.

Acceptance Criteria

• Root cause of JMT data size is investigated and documented in the issue
• A proposed optimization solution is written up with expected size reduction
• The optimization is implemented and reduces jmt_node_data storage by at least 50%
• Existing functionality (block validation, state queries, reorgs) continues to work correctly
• Benchmarks comparing before/after storage sizes are provided

Context

• Issue opened by @stringhandler
• jmt_node_data: 9,601,158 entries, 6.4 GB, avg 669 bytes, depth 4
• jmt_unique_key_data: 2,615,793 entries, 302.1 MB
• jmt_value_data: 2,615,793 entries, 231.8 MB
• Total LMDB map size: 11.6 GB; JMT tables account for ~60%
• Relevant code: base_layer/core/src/chain_storage/lmdb_db/

How to Claim

1. Comment on this issue to signal intent (courtesy, not a lock)
2. Fork the repo and do the work
3. Submit a PR that meets the acceptance criteria
4. First PR that passes review and gets merged wins the bounty
5. On acceptance, XTM payment is processed

Notes

• This touches core storage infrastructure — PR will require thorough review
• AI-assisted development is expected and encouraged
• If you get stuck, ask in Discord
• If you can't complete it within 14 days of claiming, let us know so someone else can pick it up

[SWvheerden]

Description

JMT (Jellyfish Merkle Tree) data is the single largest consumer of storage in the base node database — jmt_node_data alone accounts for 6.4 GB out of 11.4 GB total (see table in issue body). We need to investigate what is driving the excessive size and propose a solution.

Acceptance criteria

• Investigate and confirm the root cause of the JMT data size issue
• Write up a proposed optimization solution in this issue

Size: L

[metalaureate]

Original issue description (preserved before bounty rewrite)

JMT data is very large and should be optimized..

╭───────────┬───────────────────┬────────┬─────────┬──────────────────────────────────────────────────────╮
│ Component │ Database          │ Type   │ Size    │ Path                                                 │
├───────────┼───────────────────┼────────┼─────────┼──────────────────────────────────────────────────────┤
│ Base Node │ db                │ LMDB   │ 11.4 GB │ /root/.tari/mainnet/data/base_node/db                │
│ Base Node │ dht.db            │ SQLite │ 80.9 MB │ /root/.tari/mainnet/data/base_node/dht.db            │
│ Base Node │ base_node         │ LMDB   │ 58.6 MB │ /root/.tari/mainnet/peer_db/base_node                │
│ Wallet    │ wallet            │ LMDB   │ 17.7 MB │ /root/.tari/mainnet/peer_db/wallet                   │
│ Wallet    │ console_wallet.db │ SQLite │ 1.4 MB  │ /root/.tari/mainnet/data/wallet/db/console_wallet.db │
│ Wallet    │ dht.db            │ SQLite │ 53.2 KB │ /root/.tari/mainnet/data/wallet/dht.db               │
╰───────────┴───────────────────┴────────┴─────────┴──────────────────────────────────────────────────────╯

=== Detailed Base Node LMDB Analysis ===

Environment Information:
  Map Size: 11.6 GB
  Last Page: 2771729
  Last Transaction ID: 33636
  Max Readers: 126
  Used Readers: 0

Database Statistics:
╭──────────────────────────────────┬─────────┬──────────┬──────────┬───────┬─────────┬─────────┬────────┬──────────╮
│ Database                         │ Entries │ Size     │ Avg Size │ Depth │ Pages   │ Leaf    │ Branch │ Overflow │
├──────────────────────────────────┼─────────┼──────────┼──────────┼───────┼─────────┼─────────┼────────┼──────────┤
│ jmt_node_data                    │ 9601158 │ 6.4 GB   │ 669 B    │ 4     │ 1568844 │ 1556388 │ 12456  │ 0        │
│ utxos                            │ 1596182 │ 2.8 GB   │ 1.7 KB   │ 5     │ 675116  │ 656843  │ 18273  │ 0        │
│ inputs                           │ 1019864 │ 630.1 MB │ 617 B    │ 5     │ 153836  │ 149586  │ 4250   │ 0        │
│ jmt_unique_key_data              │ 2615793 │ 302.1 MB │ 115 B    │ 4     │ 73745   │ 72562   │ 1183   │ 0        │
│ txos_hash_to_index               │ 1596182 │ 277.1 MB │ 173 B    │ 4     │ 67640   │ 66581   │ 1059   │ 0        │
│ jmt_value_data                   │ 2615793 │ 231.8 MB │ 88 B     │ 4     │ 56593   │ 55891   │ 702    │ 0        │
│ kernels                          │ 379117  │ 187.0 MB │ 493 B    │ 4     │ 45655   │ 44206   │ 1449   │ 0        │
│ deleted_txo_hash_to_header_index │ 1019864 │ 176.4 MB │ 173 B    │ 4     │ 43077   │ 42513   │ 564    │ 0        │
│ kernel_excess_sig_index          │ 379117  │ 86.6 MB  │ 228 B    │ 4     │ 21141   │ 20594   │ 547    │ 0        │
│ utxo_commitment_index            │ 576318  │ 67.4 MB  │ 116 B    │ 4     │ 16447   │ 16181   │ 266    │ 0        │
│ kernel_excess_index              │ 379117  │ 66.0 MB  │ 174 B    │ 4     │ 16122   │ 15856   │ 266    │ 0        │
│ orphans                          │ 156     │ 50.8 MB  │ 325.9 KB │ 2     │ 12411   │ 19      │ 1      │ 12391    │
│ headers                          │ 20192   │ 16.2 MB  │ 801 B    │ 3     │ 3949    │ 3927    │ 19     │ 3        │
│ mmr_peak_data                    │ 20192   │ 12.1 MB  │ 599 B    │ 3     │ 2955    │ 2940    │ 15     │ 0        │
│ header_accumulated_data          │ 20192   │ 4.3 MB   │ 212 B    │ 3     │ 1046    │ 1040    │ 6      │ 0        │
│ block_hashes                     │ 20192   │ 1.5 MB   │ 74 B     │ 3     │ 367     │ 362     │ 5      │ 0        │
│ kernel_mmr_size_index            │ 20192   │ 540.7 KB │ 26 B     │ 2     │ 132     │ 131     │ 1      │ 0        │
│ orphan_accumulated_data          │ 130     │ 53.2 KB  │ 409 B    │ 2     │ 13      │ 12      │ 1      │ 0        │
│ orphan_chain_tips                │ 116     │ 20.5 KB  │ 176 B    │ 2     │ 5       │ 4       │ 1      │ 0        │
│ orphan_parent_map_index          │ 156     │ 20.5 KB  │ 131 B    │ 2     │ 5       │ 4       │ 1      │ 0        │
│ reorgs                           │ 94      │ 20.5 KB  │ 217 B    │ 2     │ 5       │ 4       │ 1      │ 0        │
│ metadata                         │ 8       │ 4.1 KB   │ 512 B    │ 1     │ 1       │ 1       │ 0      │ 0        │
│ monero_seed_height               │ 11      │ 4.1 KB   │ 372 B    │ 1     │ 1       │ 1       │ 0      │ 0        │
│ monero_seed_height_index         │ 11      │ 4.1 KB   │ 372 B    │ 1     │ 1       │ 1       │ 0      │ 0        │
│ bad_blocks                       │ 1       │ 4.1 KB   │ 4.1 KB   │ 1     │ 1       │ 1       │ 0      │ 0        │
│ unique_id_index                  │ 0       │ 0 B      │ 0 B      │ 0     │ 0       │ 0       │ 0      │ 0        │
│ contract_index                   │ 0       │ 0 B      │ 0 B      │ 0     │ 0       │ 0       │ 0      │ 0        │
│ validator_nodes                  │ 0       │ 0 B      │ 0 B      │ 0     │ 0       │ 0       │ 0      │ 0        │
│ validator_nodes_mapping          │ 0       │ 0 B      │ 0 B      │ 0     │ 0       │ 0       │ 0      │ 0        │
│ template_registrations           │ 0       │ 0 B      │ 0 B      │ 0     │ 0       │ 0       │ 0      │ 0        │
│ utxo_smt                         │ 0       │ 0 B      │ 0 B      │ 0     │ 0       │ 0       │ 0      │ 0        │
╰──────────────────────────────────┴─────────┴──────────┴──────────┴───────┴─────────┴─────────┴────────┴──────────╯

[stringhandler]

please open a new issue