ethereum/execution-spec-tests is both a collection of test cases and a framework implemented in Python to generate tests for Ethereum execution clients.
The framework collects and executes the test cases in order to generate test fixtures (JSON) which can be consumed by any execution client to verify their implementation of ethereum/execution-specs. The fixtures, which define state transition and block tests, are generated by the framework using one of the t8n command-line tools that are provided by most execution clients, see below for an overview of the supported t8n tools.
---
title: Test Fixture Generation with execution-spec-tests
---
flowchart LR
style C stroke:#333,stroke-width:2px
style D stroke:#333,stroke-width:2px
style G stroke:#F9A825,stroke-width:2px
style H stroke:#F9A825,stroke-width:2px
subgraph "ethereum/go-ethereum, ..."
C[<code>evm t8n</code>\nexternal executable]
end
subgraph ethereum/solidity
D[<code>solc</code>\nexternal executable]
end
subgraph ethereum/EIPs
E(<code>EIPS/EIP-*.md</code>\nSHA digest via Github API)
end
subgraph "ethereum/execution-spec-tests"
A(<code>./tests/**/*.py</code>\nPython Test Cases)
B([<code>$ fill ./tests/</code>\nPython Framework])
end
subgraph Test Fixture Consumers
subgraph ethereum/hive
G([<code>$ hive ...</code>\nGo Test Framework])
end
H([Client executables])
end
C <-.-> B
D <-.-> B
A --> B
E <-.-> |retrieve latest spec version\ncheck tested spec version| B
B -->|output| F(<code>./fixtures/**/*.json</code>\nJSON Test Fixtures)
F -->|input| G
F -->|input| H
The generated test fixtures can be used:
- Directly by client teams' test frameworks, and,
- In the integration tests executed in the ethereum/hive framework.
The following transition tools are supported by the framework:
| Client | "t8n" Tool | Tracing Support |
|---|---|---|
| ethereum/evmone | evmone-t8n |
Yes |
| ethereum/execution-specs | ethereum-spec-evm |
Yes |
| ethereum/go-ethereum | evm t8n |
Yes |
| ethereumjs | ethereumjs-t8ntool.sh |
No |
| hyperledger/besu | evm t8n-server |
Yes |
| status-im/nimbus-eth1 | t8n |
Yes |
Generally, specific t8n implementations and branches must be used when developing tests for upcoming EIPs.
We use named reference tags to point to the specific version of the t8n implementation that needs to be used fill the tests.
All current tags, their t8n implementation and branch they point to, are listed in .github/configs/evm.yaml.
The tools provided by ethereum/execution-spec-tests use uv (docs.astral.sh/uv) to manage their dependencies and virtual environment. uv downloads Python for your target platform if one of the required versions (Python 3.10, 3.11 or 3.12) is not available natively.
uv can be installed via curl (recommended; can self-update):
curl -LsSf https://astral.sh/uv/install.sh | shor pip (requires Python, can't self-update):
pip install uv