feat(consume): consume direct using evmone-state and blockchaintest

[pdobacz]

🗒️ Description

First step towards supporting consume direct with evmone.

Remarks:

1. Starting off with just state_test support to get the ball rolling
2. evmone has two things which make things harder. One is that it has two separate binaries: evmone-statetest and evmone-blockchaintest, rather than a single one with 2 subcommands, as EEST assumes. Decision we need to make where do we adjust - on EEST end or on evmone end. Gut tells me it is more elegant to adjust evmone, but overall easier to do so in EEST (assuming we can cut a corner and let EEST choose state/blockchain binary on the fly as it picks up fixtures). EDIT: using evmone-blockchaintest is handled in this PR 👇
3. Other is that evmone tests on the granularity level of a single json file. This means a failed test case fails the entire file, making it a bit harder to figure out where the failure is (failure message provides a good hint, but is pretty messy). Also test filtering is very coarse. Something we need to tackle on evmone end as we progress.

Opening as draft - let me know if OK to proceed with such first step into main.

🔗 Related Issues or PRs

N/A.

✅ Checklist

• All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered adding an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).

[spencer-tb]

LGTM from my side! Thanks for adding this. Feel free to covert to ready for review :)

[danceratopz]

@pdobacz wow, this is amazing!

consume direct --bin=~/bin/evmone-statetest [email protected] -v -m state_test -n 8

🥳

2. evmone has two things which make things harder. One is that it has two separate binaries: evmone-statetest and evmone-blockchaintest, rather than a single one with 2 subcommands, as EEST assumes. Decision we need to make where do we adjust - on EEST end or on evmone end.

You can specify two binaries as --bin=evmone-statetest --bin=evmone-blockchaintest, then configure the EvmoneFixtureConsumer appropriately so that it picks the relevant binary based on binary version output and fixture type.

This works right now, for example:

consume direct --bin=~/bin/evmone-statetest --bin=~/bin/evm [email protected] -v -m "state_test and Prague" -k test_chainid

Edit: Just to clarify, this parametrizes each test by Geths' evm and evmone-statetest:

I don't think we should merge this as-is. It fails by default if you provide a blockchain_test fixture. Instead, the test case should rather be skipped - then I'd be ok to merge!

[pdobacz]

Makes sense, will do! My initial plan for the EEST modification was to make it have some "leeway" in the way one specifies the binary. I.e. when evmone-statetest is specified, it would switch to evmone-blockchaintest when needed, assuming it's there (and vice versa). Could also add a version check to ensure both binaries match.

Not sure how the dual --bin works yet, but I'll take a look, thx.

I'll expand this PR soon, but I suppose the "granularity" issue (3.) can be fixed in a followup?

[danceratopz]

Makes sense, will do! My initial plan for the EEST modification was to make it have some "leeway" in the way one specifies the binary. I.e. when evmone-statetest is specified, it would switch to evmone-blockchaintest when needed, assuming it's there (and vice versa). Could also add a version check to ensure both binaries match.

Not sure how the dual --bin works yet, but I'll take a look, thx.

I think the architecture should be flexible enough as-is.

The direct plugin builds a list of FixtureConsumers:
https://github.com/ipsilon/execution-spec-tests/blob/1819482d0a0b5c6cddfd25138e22ce40eab9dfb2/src/pytest_plugins/consume/direct/conftest.py#L79-L86

The FixtureConsumer class detects what type of FixtureConsumer* subclass it should instantiate based on their -v output, luckily the two binaries have the required output:

• evmone-statetest 0.16.0-7+commit.0591d81e,
• evmone-blockchaintest 0.16.0-7+commit.0591d81e.

This allows us to define two FixtureConsumer classes for evmone:

• EvmOneStateFixtureConsumer,
• EvmOneBlockchainFixtureConsumer.

Then you use the appropriate class in consume_state_test_file() and consume_blockchain_test_file().

I'll expand this PR soon, but I suppose the "granularity" issue (3.) can be fixed in a followup?

Yes, I wouldn't let 3. block this PR!

Thanks for adding this!

[marioevz]

This allows us to define two FixtureConsumer classes for evmone:

* `EvmOneStateFixtureConsumer`,

* `EvmOneBlockchainFixtureConsumer`.

I agree on this point, I think this should be the way to go IMO.

If we implement that, and then doing --bin=evmone-statetest --bin=evmone-blockchaintest still results in fails for BlockchainTest-formatted tests, then there's something we need to fix in EEST.

Thanks for implementing this @pdobacz !

[pdobacz]

Okay, this approach works up to this point:

user@pop-os:~/sources/ethereum/execution-spec-tests$ PATH=$PATH:/home/user/sources/ethereum/evmone/build/bin/ uv run consume direct --bin=evmone-statetest --bin=evmone-blockchaintest --input /home/user/download_fixtures/fixtures_develop_v5.2.0_eip7939/fixtures/
<snip>                                                      

EvmOneFixtureConsumer-tests/osaka/eip7939_count_leading_zeros/test_count_leading_zeros.py .............................................................................................................. [ 110/2386]
........................................................................................................................................................................................................ [ 310/2386]
........................................................................................................................................................................................................ [ 510/2386]
......................................................................................ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 710/2386]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 910/2386]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [1110/2386]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss                                                                                                                      [1193/2386]
EvmOneBlockchainFixtureConsumer-tests/osaka/eip7939_count_leading_zeros/test_count_leading_zeros.py ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [1293/2386]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [1493/2386]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [1693/2386]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........................................................................................................ [1893/2386]
........................................................................................................................................................................................................ [2093/2386]
........................................................................................................................................................................................................ [2293/2386]
.............................................................................................                                                                                                            [2386/2386]
<snip>
======================================================================================== 1193 passed, 1193 skipped in 4.89s =========================================================================================

You'll notice the s-s. I think since test case discovery and consumer discovery are separate (and should stay this way), the BlockchainConsumer has no other option than pytest.skip on state fixtures and vice versa, leading to this noisy output. LMK if this works for you (or maybe there's some way to skip these cases on discovery I'm missing)

[pdobacz]

blockchaintest done with the caveat as mentioned above, ready to review.

[marioevz]

LGTM, amazing work, thanks!