|
| 1 | +# Multithreaded Cannon: Failure Modes and Recovery Path Analysis |
| 2 | + |
| 3 | +<!-- START doctoc generated TOC please keep comment here to allow auto update --> |
| 4 | +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> |
| 5 | + |
| 6 | +- [Introduction](#introduction) |
| 7 | +- [Failure Modes and Recovery Paths](#failure-modes-and-recovery-paths) |
| 8 | + - [Incorrect Linux/MIPS emulation](#incorrect-linuxmips-emulation) |
| 9 | + - [Unimplemented syscalls or opcodes needed by `op-program`](#unimplemented-syscalls-or-opcodes-needed-by-op-program) |
| 10 | + - [Insufficient memory in the program](#insufficient-memory-in-the-program) |
| 11 | + - [Compromised Control Flow in the program](#compromised-control-flow-in-the-program) |
| 12 | + - [Failure to run correct VM based on prestate input](#failure-to-run-correct-vm-based-on-prestate-input) |
| 13 | + - [Mismatch between on-chain and off-chain execution](#mismatch-between-on-chain-and-off-chain-execution) |
| 14 | + - [Livelocks in the fault proof](#livelocks-in-the-fault-proof) |
| 15 | + - [Execution traces too long for the fault proof](#execution-traces-too-long-for-the-fault-proof) |
| 16 | + - [Invalid `DisputeGameFactory.setImplementation` execution](#invalid-disputegamefactorysetimplementation-execution) |
| 17 | +- [Action Items](#action-items) |
| 18 | +- [Audit requirements](#audit-requirements) |
| 19 | + |
| 20 | +<!-- END doctoc generated TOC please keep comment here to allow auto update --> |
| 21 | + |
| 22 | +| | | |
| 23 | +|--------|--------------| |
| 24 | +| Author | Paul Dowman, Mofi Taiwo | |
| 25 | +| Created at | *2024-10-09* | |
| 26 | +| Initial Reviewers | Matt Solomon | |
| 27 | +| Need Approval From | Tom Assas | |
| 28 | +| Status | Final | |
| 29 | + |
| 30 | +> [!NOTE] |
| 31 | +> 📢 Remember: |
| 32 | +> |
| 33 | +> - The single approver in the “Need Approval From” must be from the Security team. |
| 34 | +> - Maintain the “Status” property accordingly. An FMA document can have the following statuses: |
| 35 | +> - **Draft 📝:** Doc is created but not yet ready for review. |
| 36 | +> - **In Review 🔎:** Security is reviewing, and Engineering is iterating on the design. A checklist of action items will be created during this phase. |
| 37 | +> - **Implementing Actions 🛫:** Security has signed off on the content of the document, including the resulting action items. Engineering is responsible for implementing the the action items, and updating the checklist. |
| 38 | +> - **Final 👍:** Security will transition the status of the document to Final once all action items are completed. |
| 39 | +
|
| 40 | +> [!TIP] |
| 41 | +> Guidelines for writing a good analysis, and what the reviewer will look for: |
| 42 | +> |
| 43 | +> - Show your work: Include steps and tools for each conclusion. |
| 44 | +> - Completeness of risks considered. |
| 45 | +> - Include both implementation and operational failure modes |
| 46 | +> - Provide references to support the reviewer. |
| 47 | +> - The size of the document will likely be proportional to the project's complexity. |
| 48 | +> - The ultimate goal of this document is to identify action items to improve the security of the project. The FMA review process can be accelerated by proactively identifying action items during the writing process. |
| 49 | +
|
| 50 | +## Introduction |
| 51 | + |
| 52 | +This document covers the conversion of the [Cannon Fault Proof VM](https://docs.optimism.io/stack/protocol/fault-proofs/cannon) to support multi-threading and 64-bit architecture. These changes increase addressable memory and support better memory management by unlocking garbage collection in the op-program. |
| 53 | + |
| 54 | +The multi-threaded Fault Proof VM is specified [here](https://github.com/ethereum-optimism/specs/blob/3abc17a68727e22c31a7a113be935943f717ee63/specs/experimental/cannon-fault-proof-vm-mt.md). The current version of the MT-Cannon contract can be found [here](https://github.com/ethereum-optimism/optimism/blob/68f77aaa317b9184cbbcd1526bc57bce1722906b/packages/contracts-bedrock/src/cannon/MIPS64.sol). |
| 55 | + |
| 56 | +## Failure Modes and Recovery Paths |
| 57 | + |
| 58 | +### Incorrect Linux/MIPS emulation |
| 59 | + |
| 60 | +- **Description:** An incorrectly implemented FPVM could result in an invalid fault proof. This can be caused by bugs in the thread scheduler, incorrect emulation of MIPS64 instructions, and so on. |
| 61 | +- **Risk Assessment:** High severity, Low likelihood. |
| 62 | +- **Mitigations:** Comprehensive testing. This includes full test coverage of every supported MIPS instruction, threading semantics, and verifying op-program execution on live chain data. |
| 63 | + This includes [unit and fuzz](https://github.com/ethereum-optimism/optimism/tree/eabf70498f68f321f5de003f1d443d3e3c8100b8/cannon/mipsevm/tests) testing of MIPS instructions and Linux syscalls. It also includes [testing](https://github.com/ethereum-optimism/optimism/blob/eabf70498f68f321f5de003f1d443d3e3c8100b8/cannon/mipsevm/multithreaded/state_test.go) of multithreaded specific functionality. |
| 64 | +- **Detection:** [op-dispute-mon](https://github.com/ethereum-optimism/optimism/tree/develop/op-dispute-mon#readme) forecasts and alerts on undesirable game resolutions. |
| 65 | +- **Recovery Path(s)**: See [Fault Proof Recovery](https://www.notion.so/oplabs/RB-000-Fault-Proofs-Recovery-Runbook-8dad0f1e6d4644c281b0e946c89f345f). |
| 66 | + |
| 67 | +### Unimplemented syscalls or opcodes needed by `op-program` |
| 68 | + |
| 69 | +- **Description:** We only aim to implement syscalls and opcodes that are required by `op-program` so there are some unimplemented. It's not feasible with the time and resources available to implement an entirely general purpose VM and Linux, and this would result in a lot of additional risk due to the added complexity and unused code paths. The risk is that there is some previously untested code path that uses an opcode or syscall that we haven't implemented and this code path ends up being exercised by an input condition some time in the future. |
| 70 | +- **Risk Assessment:** High severity, low likelihood. |
| 71 | +- **Mitigations:** We periodically use Cannon to execute the op-program using inputs from op-mainnet and op-sepolia. This periodic cannon runner ([vm-runner](https://github.com/ethereum-optimism/optimism/tree/develop/op-challenger/runner)) runs on oplabs infrastructure. The vm-runner samples game inputs for the latest L2 safe head every 2 hours and uses cannon to execute the op-program using the sampled inputs. Note that this sampling does not include every game created. |
| 72 | +Furthermore, we [sanitize](https://github.com/ethereum-optimism/optimism/blob/eabf70498f68f321f5de003f1d443d3e3c8100b8/cannon/Makefile#L51) the op-program [in CI](https://github.com/ethereum-optimism/optimism/blob/eabf70498f68f321f5de003f1d443d3e3c8100b8/.circleci/config.yml#L928C1-L929C111) for unsupported opcodes. |
| 73 | +- **Detection:** [Alerting](https://github.com/ethereum-optimism/k8s/blob/master/grafana-cloud/rules/challenger.yaml#L90-L110) is setup to notify the proofs team whenever the vm-runner fails to complete a cannon run. And the CI check provides an early warning against unsupported opcodes. |
| 74 | +- **Recovery Path(s)**: See [Fault Proof Recovery](https://www.notion.so/oplabs/RB-000-Fault-Proofs-Recovery-Runbook-8dad0f1e6d4644c281b0e946c89f345f). |
| 75 | + |
| 76 | +### Insufficient memory in the program |
| 77 | + |
| 78 | +- **Description:** The op-program may run out of memory, causing it to crash. |
| 79 | +- **Risk Assessment:** High severity, low likelihood. |
| 80 | +- **Mitigations:** The 64-bit address space virtually eliminates memory exhaustion risks. Go's concurrent garbage collector automatically manages memory through scheduled background goroutines. |
| 81 | +- **Detection:** - [op-dispute-mon](https://github.com/ethereum-optimism/optimism/tree/develop/op-dispute-mon#readme) forecasts and alerts on undesirable game resolutions that would result due to a program crash. |
| 82 | +- Also in case of high memory usage or potential OOM the proof team oncall would be alerted. |
| 83 | +- **Recovery Path(s)**: See [Fault Proof Recovery](https://www.notion.so/oplabs/RB-000-Fault-Proofs-Recovery-Runbook-8dad0f1e6d4644c281b0e946c89f345f). |
| 84 | + |
| 85 | +### Compromised Control Flow in the program |
| 86 | + |
| 87 | +- **Description:** This could theoretically occur when the op-program runs out of memory in a way that lets the attacker reuse code to subvert execution. |
| 88 | +- **Risk Assessment:** High severity, low likelihood. |
| 89 | + - Low likelihood: This requires an attacker to craft inputs that not only induce high memory usage, but also corrupt or spray the heap in a way that either produces invalid fault proofs or prevents valid fault proofs from being generated. |
| 90 | +- **Mitigations:** As with [Insufficient memory in the program](#insufficient-memory-in-the-program), the 64-bit address space effectively prevents this from occurring. Furthermore, the Go runtime checks memory allocations against heap corruption. However, such memory protections may not hold due to bugs in the Go runtime. Using the `unsafe` package in go can lead to unexpected behavior and we don't use it in op-program. |
| 91 | +- **Detection:** [op-dispute-mon](https://github.com/ethereum-optimism/optimism/tree/develop/op-dispute-mon#readme) forecasts and alerts on undesirable game resolutions that would result due to honest claims being disputed at the bottom of the game tree. |
| 92 | +- **Recovery Path(s)**: See [Fault Proof Recovery](https://www.notion.so/oplabs/RB-000-Fault-Proofs-Recovery-Runbook-8dad0f1e6d4644c281b0e946c89f345f). |
| 93 | + |
| 94 | +### Unknown Absolute Prestate |
| 95 | + |
| 96 | +This absolute prestate is essentially a hash of the op-program memory loaded by Cannon, including its code. |
| 97 | +As such, the absolute prestate (or simple prestate) encodes the op-program. The MT-Cannon VM loads the op-program differently from single threaded Cannon. |
| 98 | +This warrants a new absolute prestate _hash_ to properly configure the `FaultDisputeGame` implementations for the MT-Cannon VM. |
| 99 | + |
| 100 | +- **Description:** This is when there is not a known prestate (preimage) for a given absolute prestate hash in the dispute game implementations. |
| 101 | +- **Risk Assessment:** High severity, Low likelihood. |
| 102 | +- **Mitigations:** Every absolute prestate is built off of an op-program release tag. The prestate is [build is reproducible](https://github.com/ethereum-optimism/optimism/blob/68f77aaa317b9184cbbcd1526bc57bce1722906b/op-program/Dockerfile.repro) such that the same prestate is emitted regardless of the environment. |
| 103 | +Furthermore, governance and Guardian signers will be instructed to reproduce the prestate build themselves and check that the prestate hash matches the op-program release that will be referenced in the MT-Cannon governance post. |
| 104 | +Lastly, there exists a [CI check in the monorepo](https://github.com/ethereum-optimism/optimism/blob/68f77aaa317b9184cbbcd1526bc57bce1722906b/.circleci/config.yml#L1575) asserting that every op-program github release tag maps to the expected absolute prestate hash. This ensures that op-program releases stay reproducible. |
| 105 | +- **Detection:** The vm-runner is configured to use the on-chain absolute prestate for cannon executions. Any failures in the vm-runner triggers an alert to the proofs team. |
| 106 | +- **Recovery Path(s)**: See [Fault Proof Recovery](https://www.notion.so/oplabs/RB-000-Fault-Proofs-Recovery-Runbook-8dad0f1e6d4644c281b0e946c89f345f). |
| 107 | + |
| 108 | +### Failure to run correct VM based on absolute prestate input |
| 109 | + |
| 110 | +- **Description:** The off-chain version of Cannon [attempts to run the correct VM version based on the absolute prestate input](https://github.com/ethereum-optimism/design-docs/blob/0034943e42b8ab5f9dd9ded2ef2b6b55359c922c/cannon-state-versioning.md). If it doesn't work correctly the on-chain steps would not match. |
| 111 | +- **Risk Assessment:** Medium severity, low likelihood. |
| 112 | +- **Mitigations:** Multicannon mitigates this issue by embedding a variety of cannon STFs into a single binary. This shifts the concern of ensuring the correct VM selection to multicannon. We also run multicannon on oplabs infra via the vm-runner, to assert the multicannon binary was built correctly. |
| 113 | +- **Detection:** This can be detected by manual review. Failing that, it would only be detected when malicious activity occurs and an honest op-challenger fails to generate a fault proof. |
| 114 | +- **Recovery Path(s)**: Fix the op-challenger multicannon configuration. |
| 115 | + |
| 116 | +### Mismatch between on-chain and off-chain execution |
| 117 | + |
| 118 | +- **Description:** There could be bugs in the implementation of either the Solidity or Go versions that make them incompatible with each other. |
| 119 | +- **Risk Assessment:** High severity, low likelihood. |
| 120 | +- **Mitigations:** [Differential testing](https://github.com/ethereum-optimism/optimism/tree/eabf70498f68f321f5de003f1d443d3e3c8100b8/cannon/mipsevm/tests) asserts identical on-chain and off-chain execution. |
| 121 | +- **Detection:** An op-challenger fails to fault prove an invalid claim using a witness generated offchain. |
| 122 | +- **Recovery Path(s)**: Depends on the specifics. If the off-chain version is changed to match the onchain VM implementation, then fixing this can be done solely offchain. In the opposite case (changing the onchain version to match the off-chain version) a governance vote will be needed. As usual, the [Fault Proof Recovery](https://www.notion.so/oplabs/RB-000-Fault-Proofs-Recovery-Runbook-8dad0f1e6d4644c281b0e946c89f345f) provides the best guidance on this. |
| 123 | + |
| 124 | +### Livelocks in the fault proof |
| 125 | + |
| 126 | +- **Description:** A livelocked execution prevents an honest challenger from generating a fault proof. |
| 127 | +- **Risk Assessment:** High severity, low likelihood. |
| 128 | +- **Mitigations:** Manual review of the op-program and a quick review of Go runtime internals. The op-program uses 3 threads, and only one of those threads is used by the mutator main function. This makes livelocks very unlikely. We looked into the livelock problem and there are possible solutions but they're deferred for future work as the risk of a livelock is considered too low to be addressed immediately. |
| 129 | +- **Detection:** This would manifest as an execution that runs forever. Eventually, but well before the dispute period ends, op-dispute-mon will indicate that a game is forecasted to resolve incorrectly. |
| 130 | +- **Recovery Path(s)**: See [Fault Proof Recovery](https://www.notion.so/oplabs/RB-000-Fault-Proofs-Recovery-Runbook-8dad0f1e6d4644c281b0e946c89f345f). |
| 131 | + |
| 132 | +### Execution traces too long for the fault proof |
| 133 | + |
| 134 | +- **Description:** It's possible that introducing multi-threading/gc greatly increases the execution time of the op-program. |
| 135 | +- **Risk Assessment:** Medium severity, low likelihood. |
| 136 | +- **Mitigations:** Based on [vm-runner executions of 64-bit Cannon and 32-bit single-threaded Cannon](https://optimistic.grafana.net/goto/IlD3Tu4Hg?orgId=1), the 64-bit VM executes the op-program much faster than the 32-bit VM. As such, we do not expect worse performance over the existing VM, which was already found to be adequate. However, we can always use CPUs with better single-core performance to mitigate. |
| 137 | +- **Detection:** op-dispute-mon provides an early forecast (see [this alerting rule](https://github.com/ethereum-optimism/k8s/blob/6416658dd9602f83c6be093c5c57af45877258a3/grafana-cloud/rules/dispute-mon.yaml#L154)) and triggers an alert if the op-challenger stops interacting with a game. |
| 138 | +- **Recovery Path(s)**: This can be mitigated by migrating the op-challenger to a more powerful CPU. [Fault Proof Recovery](https://www.notion.so/oplabs/RB-000-Fault-Proofs-Recovery-Runbook-8dad0f1e6d4644c281b0e946c89f345f) provides guidance on when it'll be appropriate to do so. |
| 139 | + |
| 140 | +### Invalid `DisputeGameFactory.setImplementation` execution |
| 141 | + |
| 142 | +The `DisputeGameFactory.setImplementation` will be used to upgrade both the `CANNON` and `PERMISSIONED_CANNON` game types. Note that this means this upgrade will not require changing the `respectedGameType` in the `OptimismPortal` as `CANNON` games will continue to be used to finalize outputs, albeit with a different VM. |
| 143 | +As such, there will be a brief moment where there are two sets of `CANNON` games that using singlethreaded and multithreaded VMs. |
| 144 | + |
| 145 | +- Description: This occurs when either the call to the DisputeGameFactory could not be made due to grossly unfavorable base fees on L1, an invalidly approved safe nonce, or a successful execution to a misconfigured dispute game implementation. |
| 146 | +- **Risk Assessment:** Low severity, low likelihood. |
| 147 | + - Low Likelihood: The low likelihood is a result of tenderly simulation testing of safe transactions, code review of the upgrade playbook, and manual review of the dispute game implementations (which are deployed on mainnet and specified in the governance proposal so they may be reviewed). |
| 148 | + - Low severity: Fault Proofs continues to use the existing single-threaded FPVM. This carries a reputational risk, but it doesn't diminish the security of the system. Withdrawals will continue to work against outputs secured by the single-threaded FPVM. |
| 149 | +- **Mitigations:** No immediate action is needed other than to retry the safe transaction. This may require another signing ceremony. Note that the op-challenger does not need to be rolled back, as multicannon is backwards compatible with older FPVM state transition functions, since this is not a real hardfork. Thus the impact is classified as **low impact** compare to [invalid `DisputeGameFactory.setImplementation` execution during hardfork](https://github.com/ethereum-optimism/design-docs/blob/main/security/fma-generic-hardfork.md#invalid-disputegamefactorysetimplementation-execution). |
| 150 | +- **Detection:** An un-executed safe transaction is easily detectable. |
| 151 | +- **Recovery Path(s)**: Reschedule upgrade, possibly releasing new binary though without immediate urgency. |
| 152 | + |
| 153 | + |
| 154 | +## Action Items |
| 155 | + |
| 156 | +Below is what needs to be done before launch to reduce the chances of the above failure modes occurring, and to ensure they can be detected and recovered from: |
| 157 | + |
| 158 | +- [x] Third-party audit the offchain and onchain VM implementation and specification (Assignee: @inphi) |
| 159 | +- [x] [Add a healthcheck for the vm-runner (Assignee: @pauldowman)](https://github.com/ethereum-optimism/k8s/pull/5424) |
| 160 | +- [x] Add a dedicated Validation file for ensuring the AbsolutePrestate is valid here -> https://github.com/ethereum-optimism/superchain-ops/pull/673). |
| 161 | + |
| 162 | +## Audit requirements |
| 163 | + |
| 164 | +An audit of the multithreaded VM is not required per the [OP Labs Audit Framework](https://gov.optimism.io/t/op-labs-audit-framework-when-to-get-external-security-review-and-how-to-prepare-for-it/6864). |
| 165 | +A failure in the new Cannon VM and thus dispute games is mitigated by an airgap in finalized withdrawals. Furthermore, there's a window whereby the Security Council can override the results of invalid games. |
| 166 | +Nonetheless, out of an abundance of caution, it has been [audited by Spearbit](https://github.com/ethereum-optimism/optimism/pull/14218/files) and internally by Base. |
0 commit comments