Skip to content

Commit 9ab4401

Browse files
jamesobjamesob
committed
doc: add assumeutxo notes
1 parent 9013f23 commit 9ab4401

File tree

2 files changed

+139
-0
lines changed

2 files changed

+139
-0
lines changed

‎doc/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ The Bitcoin repo's [root README](/README.md) contains relevant information on th
7171

7272
### Miscellaneous
7373
- [Assets Attribution](assets-attribution.md)
74+
- [Assumeutxo design](assumeutxo.md)
7475
- [bitcoin.conf Configuration File](bitcoin-conf.md)
7576
- [Files](files.md)
7677
- [Fuzz-testing](fuzzing.md)

‎doc/assumeutxo.md

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# assumeutxo
2+
3+
Assumeutxo is a feature that allows fast bootstrapping of a validating bitcoind
4+
instance with a very similar security model to assumevalid.
5+
6+
The RPC commands `dumptxoutset` and `loadtxoutset` are used to respectively generate
7+
and load UTXO snapshots. The utility script `./contrib/devtools/utxo_snapshot.sh` may
8+
be of use.
9+
10+
## General background
11+
12+
- [assumeutxo proposal](https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal)
13+
- [Github issue](https://github.com/bitcoin/bitcoin/issues/15605)
14+
- [draft PR](https://github.com/bitcoin/bitcoin/pull/15606)
15+
16+
## Design notes
17+
18+
- A new block index `nStatus` flag is introduced, `BLOCK_ASSUMED_VALID`, to mark block
19+
index entries that are required to be assumed-valid by a chainstate created
20+
from a UTXO snapshot. This flag is mostly used as a way to modify certain
21+
CheckBlockIndex() logic to account for index entries that are pending validation by a
22+
chainstate running asynchronously in the background. We also use this flag to control
23+
which index entries are added to setBlockIndexCandidates during LoadBlockIndex().
24+
25+
- Indexing implementations via BaseIndex can no longer assume that indexation happens
26+
sequentially, since background validation chainstates can submit BlockConnected
27+
events out of order with the active chain.
28+
29+
- The concept of UTXO snapshots is treated as an implementation detail that lives
30+
behind the ChainstateManager interface. The external presentation of the changes
31+
required to facilitate the use of UTXO snapshots is the understanding that there are
32+
now certain regions of the chain that can be temporarily assumed to be valid (using
33+
the nStatus flag mentioned above). In certain cases, e.g. wallet rescanning, this is
34+
very similar to dealing with a pruned chain.
35+
36+
Logic outside ChainstateManager should try not to know about snapshots, instead
37+
preferring to work in terms of more general states like assumed-valid.
38+
39+
40+
## Chainstate phases
41+
42+
Chainstate within the system goes through a number of phases when UTXO snapshots are
43+
used, as managed by `ChainstateManager`. At various points there can be multiple
44+
`CChainState` objects in existence to facilitate both maintaining the network tip and
45+
performing historical validation of the assumed-valid chain.
46+
47+
It is worth noting that though there are multiple separate chainstates, those
48+
chainstates share use of a common block index (i.e. they hold the same `BlockManager`
49+
reference).
50+
51+
The subheadings below outline the phases and the corresponding changes to chainstate
52+
data.
53+
54+
### "Normal" operation via initial block download
55+
56+
`ChainstateManager` manages a single CChainState object, for which
57+
`m_snapshot_blockhash` is null. This chainstate is (maybe obviously)
58+
considered active. This is the "traditional" mode of operation for bitcoind.
59+
60+
| | |
61+
| ---------- | ----------- |
62+
| number of chainstates | 1 |
63+
| active chainstate | ibd |
64+
65+
### User loads a UTXO snapshot via `loadtxoutset` RPC
66+
67+
`ChainstateManager` initializes a new chainstate (see `ActivateSnapshot()`) to load the
68+
snapshot contents into. During snapshot load and validation (see
69+
`PopulateAndValidateSnapshot()`), the new chainstate is not considered active and the
70+
original chainstate remains in use as active.
71+
72+
| | |
73+
| ---------- | ----------- |
74+
| number of chainstates | 2 |
75+
| active chainstate | ibd |
76+
77+
Once the snapshot chainstate is loaded and validated, it is promoted to active
78+
chainstate and a sync to tip begins. A new chainstate directory is created in the
79+
datadir for the snapshot chainstate called
80+
`chainstate_[SHA256 blockhash of snapshot base block]`.
81+
82+
| | |
83+
| ---------- | ----------- |
84+
| number of chainstates | 2 |
85+
| active chainstate | snapshot |
86+
87+
The snapshot begins to sync to tip from its base block, technically in parallel with
88+
the original chainstate, but it is given priority during block download and is
89+
allocated most of the cache (see `MaybeRebalanceCaches()` and usages) as our chief
90+
consideration is getting to network tip.
91+
92+
**Failure consideration:** if shutdown happens at any point during this phase, both
93+
chainstates will be detected during the next init and the process will resume.
94+
95+
### Snapshot chainstate hits network tip
96+
97+
Once the snapshot chainstate leaves IBD, caches are rebalanced
98+
(via `MaybeRebalanceCaches()` in `ActivateBestChain()`) and more cache is given
99+
to the background chainstate, which is responsible for doing full validation of the
100+
assumed-valid parts of the chain.
101+
102+
**Note:** at this point, ValidationInterface callbacks will be coming in from both
103+
chainstates. Considerations here must be made for indexing, which may no longer be happening
104+
sequentially.
105+
106+
### Background chainstate hits snapshot base block
107+
108+
Once the tip of the background chainstate hits the base block of the snapshot
109+
chainstate, we stop use of the background chainstate by setting `m_stop_use` (not yet
110+
committed - see #15606), in `CompleteSnapshotValidation()`, which is checked in
111+
`ActivateBestChain()`). We hash the background chainstate's UTXO set contents and
112+
ensure it matches the compiled value in `CMainParams::m_assumeutxo_data`.
113+
114+
The background chainstate data lingers on disk until shutdown, when in
115+
`ChainstateManager::Reset()`, the background chainstate is cleaned up with
116+
`ValidatedSnapshotShutdownCleanup()`, which renames the `chainstate_[hash]` datadir as
117+
`chainstate`.
118+
119+
| | |
120+
| ---------- | ----------- |
121+
| number of chainstates | 2 (ibd has `m_stop_use=true`) |
122+
| active chainstate | snapshot |
123+
124+
**Failure consideration:** if bitcoind unexpectedly halts after `m_stop_use` is set on
125+
the background chainstate but before `CompleteSnapshotValidation()` can finish, the
126+
need to complete snapshot validation will be detected on subsequent init by
127+
`ChainstateManager::CheckForUncleanShutdown()`.
128+
129+
### Bitcoind restarts sometime after snapshot validation has completed
130+
131+
When bitcoind initializes again, what began as the snapshot chainstate is now
132+
indistinguishable from a chainstate that has been built from the traditional IBD
133+
process, and will be initialized as such.
134+
135+
| | |
136+
| ---------- | ----------- |
137+
| number of chainstates | 1 |
138+
| active chainstate | ibd |

0 commit comments

Comments
 (0)