Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: limit bundle by data size #188

Open
wants to merge 44 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
d29d2ba
add test for query
segfault-magnet Jan 27, 2025
16dd0df
fix tests
segfault-magnet Jan 27, 2025
461c24e
tests passing
segfault-magnet Jan 28, 2025
cef7a89
configured via fragments per bundle
segfault-magnet Jan 28, 2025
271115b
simplify query
segfault-magnet Jan 28, 2025
36541a7
human readable logs
segfault-magnet Jan 28, 2025
f3160c4
count all available unbundled blocks when deciding whether to bundle
segfault-magnet Jan 28, 2025
035432a
add todo
segfault-magnet Jan 28, 2025
67bb02b
another todo
segfault-magnet Jan 28, 2025
f453438
don't use a db tx
segfault-magnet Jan 28, 2025
fc5ded7
add todo
segfault-magnet Jan 29, 2025
8d6182d
add attempt elimination based on attempts that failed to be under the
segfault-magnet Jan 29, 2025
54d0151
correct counting of attempts, log the max upper block limit
segfault-magnet Jan 29, 2025
fc79c67
print upper limit
segfault-magnet Jan 29, 2025
1fa0479
stop streaming as soon as a hole is detected
segfault-magnet Jan 30, 2025
07d9d41
add todo
segfault-magnet Feb 4, 2025
263f65a
Merge remote-tracking branch 'origin/master' into feat/limit_bundle_b…
segfault-magnet Feb 5, 2025
3d4a9b9
read env variable as human formatted bytes
segfault-magnet Feb 5, 2025
ed2d7cb
update config docs
segfault-magnet Feb 5, 2025
4530683
bump version because current one is out of sync from what was published
segfault-magnet Feb 5, 2025
e0141c2
return check for block_height_lookback
segfault-magnet Feb 5, 2025
abda522
add human readable values in e2e setup
segfault-magnet Feb 5, 2025
f165213
cleanup of postgre impl
segfault-magnet Feb 5, 2025
a330f4b
update comment
segfault-magnet Feb 5, 2025
ad59b12
remove comment
segfault-magnet Feb 5, 2025
c6c3946
updated comments
segfault-magnet Feb 5, 2025
91388d2
nits
segfault-magnet Feb 5, 2025
7294c97
fix typo
segfault-magnet Feb 5, 2025
c141139
add debugging info
segfault-magnet Feb 12, 2025
8b0ef58
add debug data
segfault-magnet Feb 12, 2025
b79758e
wip
segfault-magnet Feb 12, 2025
1bf0559
import chunking by num and size of blocks
segfault-magnet Feb 12, 2025
bbebfa3
import batches to internal config, tests for block batching in importer
segfault-magnet Feb 12, 2025
a62c6a7
refactor
segfault-magnet Feb 12, 2025
35c1730
cleanup the extension
segfault-magnet Feb 12, 2025
9c8bfbc
add test for single item for chunking
segfault-magnet Feb 12, 2025
f8c0d3e
fix too large vector allocation, tests
segfault-magnet Feb 12, 2025
bd2a30a
fix clippy
segfault-magnet Feb 12, 2025
1484265
remove todo
segfault-magnet Feb 12, 2025
1b74a79
add comment
segfault-magnet Feb 13, 2025
6b6bf1b
wait for costs
segfault-magnet Feb 13, 2025
b5b2775
capture table sizes over time
segfault-magnet Feb 14, 2025
426f9b4
Merge branch 'master' into feat/limit_bundle_by_data_size
segfault-magnet Feb 17, 2025
e12b39b
Merge remote-tracking branch 'origin/master' into feat/limit_bundle_b…
segfault-magnet Feb 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

52 changes: 41 additions & 11 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ members = [
]

[workspace.package]
version = "0.10.5"
version = "0.10.6"
authors = ["Fuel Labs <[email protected]>"]
edition = "2021"
homepage = "https://fuel.network/"
Expand Down Expand Up @@ -63,6 +63,7 @@ fuel-crypto = { version = "0.58.2", default-features = false }
futures = { version = "0.3", default-features = false }
hex = { version = "0.4", default-features = false }
humantime = { version = "2.1", default-features = false }
byte-unit = { version = "5.1", default-features = false }
itertools = { version = "0.13", default-features = false }
mockall = { version = "0.12", default-features = false }
num-bigint = { version = "0.4.6", default-features = false }
Expand Down
47 changes: 33 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,35 +221,36 @@ The Fuel Block Committer is configured primarily through environment variables.

- **`COMMITTER__APP__FEE_ALGO_SHORT_SMA_BLOCKS`**

- **Description:** Short-term period for the fee algo in block numbers.
- **Description:** Short-term period for the fee algorithm in block numbers.
- **Type:** Positive integer (`NonZeroU64`)
- **Example:** `25`

- **`COMMITTER__APP__FEE_ALGO_LONG_SMA_BLOCKS`**

- **Description:** Long-term period for the fee algo in block numbers.
- **Description:** Long-term period for the fee algorithm in block numbers.
- **Type:** Positive integer (`NonZeroU64`)
- **Example:** `300`

- **`COMMITTER__APP__FEE_ALGO_MAX_L2_BLOCKS_BEHIND`**

- **Description:** Maximum number of unposted L2 blocks before sending a transaction regardless of fees.
- **Description:** Maximum number of unposted L2 blocks before forcing a transaction regardless of fees.
- **Type:** Positive integer (`NonZeroU32`)
- **Example:** `28800`

- **`COMMITTER__APP__FEE_ALGO_START_MAX_FEE_MULTIPLIER`**

- **Description:** Starting multiplier applied when we're 0 L2 blocks behind.
- **Description:** Starting fee multiplier applied when the system is fully up to date (i.e., 0 L2 blocks behind).
- **Type:** `f64`
- **Example:** `0.800000`

- **`COMMITTER__APP__FEE_ALGO_END_MAX_FEE_MULTIPLIER`**

- **Description:** Ending multiplier applied if we're `max_l2_blocks_behind - 1` blocks behind.
- **Description:** Ending fee multiplier applied when the system is nearly at the maximum allowed lag (i.e., `max_l2_blocks_behind - 1`).
- **Type:** `f64`
- **Example:** `1.200000`

- **`COMMITTER__APP__FEE_ALGO_ALWAYS_ACCEPTABLE_FEE`**

- **Description:** A fee that is always acceptable regardless of other conditions.
- **Type:** `u64`
- **Example:** `1000000000000000`
Expand All @@ -262,12 +263,24 @@ The Fuel Block Committer is configured primarily through environment variables.
- **Format:** Human-readable duration
- **Example:** `30s`

- **`COMMITTER__APP__BUNDLE__BYTES_TO_ACCUMULATE`**

- **Description:** Byte threshold that, if reached before the accumulation timeout expires, will trigger bundling immediately.
- **Format:** Human-readable byte string (e.g., `10MB`, `500KB`)
- **Example:** `1MB`

- **`COMMITTER__APP__BUNDLE__BLOCKS_TO_ACCUMULATE`**

- **Description:** Number of Fuel blocks to accumulate before initiating the bundling process.
- **Type:** Positive integer (`NonZeroUsize`)
- **Example:** `5`

- **`COMMITTER__APP__BUNDLE__MAX_FRAGMENTS_PER_BUNDLE`**

- **Description:** Maximum number of fragments allowed per bundle. This limits the size of any individual bundle.
- **Type:** Positive integer (`NonZeroUsize`)
- **Example:** `10`

- **`COMMITTER__APP__BUNDLE__OPTIMIZATION_TIMEOUT`**

- **Description:** Maximum duration allocated for determining the optimal bundle size.
Expand All @@ -276,7 +289,7 @@ The Fuel Block Committer is configured primarily through environment variables.

- **`COMMITTER__APP__BUNDLE__OPTIMIZATION_STEP`**

- **Description:** Size of the optimization step at the start of the optimization process.
- **Description:** Step size used during the optimization search for the ideal bundle size.
- **Type:** Positive integer (`NonZeroUsize`)
- **Example:** `100`

Expand All @@ -285,6 +298,7 @@ The Fuel Block Committer is configured primarily through environment variables.
- **Description:** Number of fragments to accumulate before submitting them in a transaction to L1.
- **Type:** Positive integer (`NonZeroUsize`)
- **Example:** `6`
- **Note:** This value must be less than or equal to 6.

- **`COMMITTER__APP__BUNDLE__FRAGMENT_ACCUMULATION_TIMEOUT`**

Expand All @@ -294,30 +308,35 @@ The Fuel Block Committer is configured primarily through environment variables.

- **`COMMITTER__APP__BUNDLE__NEW_BUNDLE_CHECK_INTERVAL`**

- **Description:** Duration to wait before checking if a new bundle can be made.
- **Description:** Duration to wait before checking if a new bundle can be created.
- **Format:** Human-readable duration
- **Example:** `15s`

- **`COMMITTER__APP__BUNDLE__BLOCK_HEIGHT_LOOKBACK`**

- **Description:** The lookback window (in block numbers) used to determine which blocks are eligible for bundling. Blocks outside this window will be ignored.
- **Type:** `u32`
- **Example:** `100`

- **`COMMITTER__APP__BUNDLE__COMPRESSION_LEVEL`**

- **Description:** Compression level used for compressing block data before submission.
- **Values:** `"disabled"`, `"min"`, `"level1"`..`"level9"`, `"max"`
- **Allowed Values:** `"disabled"`, `"min"`, or any value up to `"max"` (`"level1"`, `"level2"`, ...)
- **Example:** `"min"`

### Configuration Validation

The committer performs validation on the provided configuration to ensure consistency and correctness. For example:
At startup, the committer validates the provided configuration to ensure that:

- **Wallet Keys:** The main wallet key and blob pool wallet key must be different.
- **Fragments to Accumulate:** Must be less than or equal to 6.
- **Block Height Lookback:** Must be greater than or equal to the number of blocks to accumulate.
- **Fee Multiplier Range:** Must have valid start and end multipliers.
- **Fragments to Accumulate:** The number of fragments to accumulate is checked to be less than or equal to 6.
- **Fee Algorithm Settings:** The fee multiplier range (start and end multipliers) is verified to be valid.

If any validation fails, the committer will return an error, preventing it from running with invalid settings.
If any validation fails, the committer will exit with an error message, preventing it from running with invalid settings.

## Running the Fee Algo Simulator

The **Fee Algo Simulator** is a separate binary designed to simulate and analyze fee algorithms using data from eth mainnet.
The **Fee Algo Simulator** is a separate binary designed to simulate and analyze fee algorithms using data from Ethereum mainnet.

### Running the Simulator

Expand Down
1 change: 1 addition & 0 deletions committer/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ rust-version = { workspace = true }

[dependencies]
actix-web = { workspace = true, features = ["macros"] }
byte-unit = { workspace = true, features = ["byte", "u128"] }
clap = { workspace = true, features = ["default", "derive"] }
clock = { workspace = true }
config = { workspace = true, features = ["async"] }
Expand Down
Loading
Loading