Skip to content

Commit 9fdd53d

Browse files
Hierarchical state diffs (#5978)
* Start extracting freezer changes for tree-states * Remove unused config args * Add comments * Remove unwraps * Subjective more clear implementation * Clean up hdiff * Update xdelta3 * Tree states archive metrics (#6040) * Add store cache size metrics * Add compress timer metrics * Add diff apply compute timer metrics * Add diff buffer cache hit metrics * Add hdiff buffer load times * Add blocks replayed metric * Move metrics to store * Future proof some metrics --------- Co-authored-by: Michael Sproul <[email protected]> * Port and clean up forwards iterator changes * Add and polish hierarchy-config flag * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Cleaner errors * Fix beacon_chain test compilation * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Patch a few more freezer block roots * Fix genesis block root bug * Fix test failing due to pending updates * Beacon chain tests passing * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Fix doc lint * Implement DB schema upgrade for hierarchical state diffs (#6193) * DB upgrade * Add flag * Delete RestorePointHash * Update docs * Update docs * Implement hierarchical state diffs config migration (#6245) * Implement hierarchical state diffs config migration * Review PR * Remove TODO * Set CURRENT_SCHEMA_VERSION correctly * Fix genesis state loading * Re-delete some PartialBeaconState stuff --------- Co-authored-by: Michael Sproul <[email protected]> * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Fix test compilation * Update schema downgrade test * Fix tests * Fix null anchor migration * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Fix tree states upgrade migration (#6328) * Towards crash safety * Fix compilation * Move cold summaries and state roots to new columns * Rename StateRoots chunked field * Update prune states * Clean hdiff CLI flag and metrics * Fix "staged reconstruction" * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Fix alloy issues * Fix staged reconstruction logic * Prevent weird slot drift * Remove "allow" flag * Update CLI help * Remove FIXME about downgrade * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Remove some unnecessary error variants * Fix new test * Tree states archive - review comments and metrics (#6386) * Review PR comments and metrics * Comments * Add anchor metrics * drop prev comment * Update metadata.rs * Apply suggestions from code review --------- Co-authored-by: Michael Sproul <[email protected]> * Update beacon_node/store/src/hot_cold_store.rs Co-authored-by: Lion - dapplion <[email protected]> * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Clarify comment and remove anchor_slot garbage * Simplify database anchor (#6397) * Simplify database anchor * Update beacon_node/store/src/reconstruct.rs * Add migration for anchor * Fix and simplify light_client store tests * Fix incompatible config test * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * More metrics * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * New historic state cache (#6475) * New historic state cache * Add more metrics * State cache hit rate metrics * Fix store metrics * More logs and metrics * Fix logger * Ensure cached states have built caches :O * Replay blocks in preference to diffing * Two separate caches * Distribute cache build time to next slot * Re-plumb historic-state-cache flag * Clean up metrics * Update book * Update beacon_node/store/src/hdiff.rs Co-authored-by: Lion - dapplion <[email protected]> * Update beacon_node/store/src/historic_state_cache.rs Co-authored-by: Lion - dapplion <[email protected]> --------- Co-authored-by: Lion - dapplion <[email protected]> * Update database docs * Update diagram * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Update lockbud to work with bindgen/etc * Correct pkg name for Debian * Remove vestigial epochs_per_state_diff * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Markdown lint * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Address Jimmy's review comments * Simplify ReplayFrom case * Fix and document genesis_state_root * Typo Co-authored-by: Jimmy Chen <[email protected]> * Merge branch 'unstable' into tree-states-archive * Compute diff of validators list manually (#6556) * Split hdiff computation * Dedicated logic for historical roots and summaries * Benchmark against real states * Mutated source? * Version the hdiff * Add lighthouse DB config for hierarchy exponents * Tidy up hierarchy exponents flag * Apply suggestions from code review Co-authored-by: Michael Sproul <[email protected]> * Address PR review * Remove hardcoded paths in benchmarks * Delete unused function in benches * lint --------- Co-authored-by: Michael Sproul <[email protected]> * Test hdiff binary format stability (#6585) * Merge remote-tracking branch 'origin/unstable' into tree-states-archive * Add deprecation warning for SPRP * Update xdelta to get rid of duplicate deps * Document test
1 parent 654fc6a commit 9fdd53d

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

57 files changed

+3350
-1681
lines changed

.github/workflows/test-suite.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -63,8 +63,8 @@ jobs:
6363
- name: Checkout repository
6464
uses: actions/checkout@v3
6565
- name: Install dependencies
66-
run: apt update && apt install -y cmake
67-
- name: Generate code coverage
66+
run: apt update && apt install -y cmake libclang-dev
67+
- name: Check for deadlocks
6868
run: |
6969
cargo lockbud -k deadlock -b -l tokio_util
7070

Cargo.lock

+76-2
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

+2
Original file line numberDiff line numberDiff line change
@@ -263,6 +263,8 @@ validator_http_metrics = { path = "validator_client/http_metrics" }
263263
validator_metrics = { path = "validator_client/validator_metrics" }
264264
validator_store= { path = "validator_client/validator_store" }
265265
warp_utils = { path = "common/warp_utils" }
266+
xdelta3 = { git = "http://github.com/sigp/xdelta3-rs", rev = "50d63cdf1878e5cf3538e9aae5eed34a22c64e4a" }
267+
zstd = "0.13"
266268

267269
[profile.maxperf]
268270
inherits = "release"

beacon_node/beacon_chain/src/beacon_chain.rs

+10-14
Original file line numberDiff line numberDiff line change
@@ -767,7 +767,6 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
767767
start_slot,
768768
local_head.beacon_state.clone(),
769769
local_head.beacon_block_root,
770-
&self.spec,
771770
)?;
772771

773772
Ok(iter.map(|result| result.map_err(Into::into)))
@@ -790,12 +789,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
790789
}
791790

792791
self.with_head(move |head| {
793-
let iter = self.store.forwards_block_roots_iterator_until(
794-
start_slot,
795-
end_slot,
796-
|| Ok((head.beacon_state.clone(), head.beacon_block_root)),
797-
&self.spec,
798-
)?;
792+
let iter =
793+
self.store
794+
.forwards_block_roots_iterator_until(start_slot, end_slot, || {
795+
Ok((head.beacon_state.clone(), head.beacon_block_root))
796+
})?;
799797
Ok(iter
800798
.map(|result| result.map_err(Into::into))
801799
.take_while(move |result| {
@@ -865,7 +863,6 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
865863
start_slot,
866864
local_head.beacon_state_root(),
867865
local_head.beacon_state.clone(),
868-
&self.spec,
869866
)?;
870867

871868
Ok(iter.map(|result| result.map_err(Into::into)))
@@ -882,12 +879,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
882879
end_slot: Slot,
883880
) -> Result<impl Iterator<Item = Result<(Hash256, Slot), Error>> + '_, Error> {
884881
self.with_head(move |head| {
885-
let iter = self.store.forwards_state_roots_iterator_until(
886-
start_slot,
887-
end_slot,
888-
|| Ok((head.beacon_state.clone(), head.beacon_state_root())),
889-
&self.spec,
890-
)?;
882+
let iter =
883+
self.store
884+
.forwards_state_roots_iterator_until(start_slot, end_slot, || {
885+
Ok((head.beacon_state.clone(), head.beacon_state_root()))
886+
})?;
891887
Ok(iter
892888
.map(|result| result.map_err(Into::into))
893889
.take_while(move |result| {

beacon_node/beacon_chain/src/block_verification.rs

-19
Original file line numberDiff line numberDiff line change
@@ -839,9 +839,6 @@ impl<T: BeaconChainTypes> GossipVerifiedBlock<T> {
839839

840840
let block_root = get_block_header_root(block_header);
841841

842-
// Disallow blocks that conflict with the anchor (weak subjectivity checkpoint), if any.
843-
check_block_against_anchor_slot(block.message(), chain)?;
844-
845842
// Do not gossip a block from a finalized slot.
846843
check_block_against_finalized_slot(block.message(), block_root, chain)?;
847844

@@ -1074,9 +1071,6 @@ impl<T: BeaconChainTypes> SignatureVerifiedBlock<T> {
10741071
.fork_name(&chain.spec)
10751072
.map_err(BlockError::InconsistentFork)?;
10761073

1077-
// Check the anchor slot before loading the parent, to avoid spurious lookups.
1078-
check_block_against_anchor_slot(block.message(), chain)?;
1079-
10801074
let (mut parent, block) = load_parent(block, chain)?;
10811075

10821076
let state = cheap_state_advance_to_obtain_committees::<_, BlockError>(
@@ -1688,19 +1682,6 @@ impl<T: BeaconChainTypes> ExecutionPendingBlock<T> {
16881682
}
16891683
}
16901684

1691-
/// Returns `Ok(())` if the block's slot is greater than the anchor block's slot (if any).
1692-
fn check_block_against_anchor_slot<T: BeaconChainTypes>(
1693-
block: BeaconBlockRef<'_, T::EthSpec>,
1694-
chain: &BeaconChain<T>,
1695-
) -> Result<(), BlockError> {
1696-
if let Some(anchor_slot) = chain.store.get_anchor_slot() {
1697-
if block.slot() <= anchor_slot {
1698-
return Err(BlockError::WeakSubjectivityConflict);
1699-
}
1700-
}
1701-
Ok(())
1702-
}
1703-
17041685
/// Returns `Ok(())` if the block is later than the finalized slot on `chain`.
17051686
///
17061687
/// Returns an error if the block is earlier or equal to the finalized slot, or there was an error

beacon_node/beacon_chain/src/builder.rs

+4
Original file line numberDiff line numberDiff line change
@@ -363,6 +363,10 @@ where
363363
store
364364
.put_block(&beacon_block_root, beacon_block.clone())
365365
.map_err(|e| format!("Failed to store genesis block: {:?}", e))?;
366+
store
367+
.store_frozen_block_root_at_skip_slots(Slot::new(0), Slot::new(1), beacon_block_root)
368+
.and_then(|ops| store.cold_db.do_atomically(ops))
369+
.map_err(|e| format!("Failed to store genesis block root: {e:?}"))?;
366370

367371
// Store the genesis block under the `ZERO_HASH` key.
368372
store

beacon_node/beacon_chain/src/historical_blocks.rs

+14-16
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ use std::iter;
1111
use std::time::Duration;
1212
use store::metadata::DataColumnInfo;
1313
use store::{
14-
chunked_vector::BlockRoots, AnchorInfo, BlobInfo, ChunkWriter, Error as StoreError,
15-
KeyValueStore,
14+
get_key_for_col, AnchorInfo, BlobInfo, DBColumn, Error as StoreError, KeyValueStore,
15+
KeyValueStoreOp,
1616
};
1717
use strum::IntoStaticStr;
1818
use types::{FixedBytesExtended, Hash256, Slot};
@@ -35,8 +35,6 @@ pub enum HistoricalBlockError {
3535
InvalidSignature,
3636
/// Transitory error, caller should retry with the same blocks.
3737
ValidatorPubkeyCacheTimeout,
38-
/// No historical sync needed.
39-
NoAnchorInfo,
4038
/// Logic error: should never occur.
4139
IndexOutOfBounds,
4240
/// Internal store error
@@ -72,10 +70,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
7270
&self,
7371
mut blocks: Vec<AvailableBlock<T::EthSpec>>,
7472
) -> Result<usize, HistoricalBlockError> {
75-
let anchor_info = self
76-
.store
77-
.get_anchor_info()
78-
.ok_or(HistoricalBlockError::NoAnchorInfo)?;
73+
let anchor_info = self.store.get_anchor_info();
7974
let blob_info = self.store.get_blob_info();
8075
let data_column_info = self.store.get_data_column_info();
8176

@@ -119,8 +114,6 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
119114

120115
let mut expected_block_root = anchor_info.oldest_block_parent;
121116
let mut prev_block_slot = anchor_info.oldest_block_slot;
122-
let mut chunk_writer =
123-
ChunkWriter::<BlockRoots, _, _>::new(&self.store.cold_db, prev_block_slot.as_usize())?;
124117
let mut new_oldest_blob_slot = blob_info.oldest_blob_slot;
125118
let mut new_oldest_data_column_slot = data_column_info.oldest_data_column_slot;
126119

@@ -158,8 +151,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
158151
}
159152

160153
// Store block roots, including at all skip slots in the freezer DB.
161-
for slot in (block.slot().as_usize()..prev_block_slot.as_usize()).rev() {
162-
chunk_writer.set(slot, block_root, &mut cold_batch)?;
154+
for slot in (block.slot().as_u64()..prev_block_slot.as_u64()).rev() {
155+
cold_batch.push(KeyValueStoreOp::PutKeyValue(
156+
get_key_for_col(DBColumn::BeaconBlockRoots.into(), &slot.to_be_bytes()),
157+
block_root.as_slice().to_vec(),
158+
));
163159
}
164160

165161
prev_block_slot = block.slot();
@@ -171,15 +167,17 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
171167
// completion.
172168
if expected_block_root == self.genesis_block_root {
173169
let genesis_slot = self.spec.genesis_slot;
174-
for slot in genesis_slot.as_usize()..prev_block_slot.as_usize() {
175-
chunk_writer.set(slot, self.genesis_block_root, &mut cold_batch)?;
170+
for slot in genesis_slot.as_u64()..prev_block_slot.as_u64() {
171+
cold_batch.push(KeyValueStoreOp::PutKeyValue(
172+
get_key_for_col(DBColumn::BeaconBlockRoots.into(), &slot.to_be_bytes()),
173+
self.genesis_block_root.as_slice().to_vec(),
174+
));
176175
}
177176
prev_block_slot = genesis_slot;
178177
expected_block_root = Hash256::zero();
179178
break;
180179
}
181180
}
182-
chunk_writer.write(&mut cold_batch)?;
183181
// these were pushed in reverse order so we reverse again
184182
signed_blocks.reverse();
185183

@@ -271,7 +269,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
271269
let backfill_complete = new_anchor.block_backfill_complete(self.genesis_backfill_slot);
272270
anchor_and_blob_batch.push(
273271
self.store
274-
.compare_and_set_anchor_info(Some(anchor_info), Some(new_anchor))?,
272+
.compare_and_set_anchor_info(anchor_info, new_anchor)?,
275273
);
276274
self.store.hot_db.do_atomically(anchor_and_blob_batch)?;
277275

beacon_node/beacon_chain/src/metrics.rs

+3
Original file line numberDiff line numberDiff line change
@@ -2004,6 +2004,7 @@ pub fn scrape_for_metrics<T: BeaconChainTypes>(beacon_chain: &BeaconChain<T>) {
20042004
let attestation_stats = beacon_chain.op_pool.attestation_stats();
20052005
let chain_metrics = beacon_chain.metrics();
20062006

2007+
// Kept duplicated for backwards compatibility
20072008
set_gauge_by_usize(
20082009
&BLOCK_PROCESSING_SNAPSHOT_CACHE_SIZE,
20092010
beacon_chain.store.state_cache_len(),
@@ -2067,6 +2068,8 @@ pub fn scrape_for_metrics<T: BeaconChainTypes>(beacon_chain: &BeaconChain<T>) {
20672068
.canonical_head
20682069
.fork_choice_read_lock()
20692070
.scrape_for_metrics();
2071+
2072+
beacon_chain.store.register_metrics();
20702073
}
20712074

20722075
/// Scrape the given `state` assuming it's the head state, updating the `DEFAULT_REGISTRY`.

0 commit comments

Comments
 (0)