This is a collection of pipelines that are built using cherry.
All data is stored in ClickHouse.
This project is meant to be run with Python 3.12
If you are using uv for development it should pick this up automatically because of the .python-version in the project root.
The docker image is configured to use this version of Python as well.
Use the main script to run a pipeline:
uv run scripts/main.pyIt takes these parameters as environment variables:
CHERRY_PIPELINE_KIND, "evm" or "svm".CHERRY_PIPELINE_NAME, name of the pipeline to run e.g. "erc20_transfers".CHERRY_FROM_BLOCK, specify the block that the indexing should start from. defaults to 0.CHERRY_TO_BLOCK, specify the block that the indexing should stop at. has no default. Indexing waits for new blocks when it reaches the tip of the chain if this argument is left empty.CHERRY_EVM_PROVIDER_KIND, specify which provider to use when indexing evm chains. Can behypersyncorsqd. Has no default and is required when indexing evm.CHERRY_EVM_CHAIN_ID, specify the chain_id when indexing an evm chain. has no default and is required when indexing evm.CHERRY_PROVIDER_BUFFER_SIZE, specify buffering between ingestion - processing - writer. Increasing this parameter might improve performance but can also cause higher memory usage. Defaults to 2.CHERRY_INIT_DB, It runs db setup script instead of the pipeline script if this is set to "true".CLICKHOUSE_HOST, defaults to127.0.0.1.CLICKHOUSE_PORT, defaults to8123.CLICKHOUSE_USER, defaults todefault.CLICKHOUSE_PASSWORD, defaults to empty string,RUST_LOGas explained in env-logger docsPY_LOGas explained in python logging docs. Defaults to "INFO"
An .env file placed in the project root can be used to define these for development.
We publish a docker image that runs the main script.
Run the docker-compose file to start a clickhouse instance for development.
docker-compose up -dRun this to delete the data on disk:
docker-compose down -vAnd this to stop the container without deleting the data:
docker-compose downThis repo uses uv for development.
- Format the code with
uv run ruff format - Lint the code with
uv run ruff check - Run type checks with
uv run pyright - Run the tests with
uv run pytest
All svm pipelines use SQD.
All evm pipelines are configurable using the CHERRY_EVM_PROVIDER_KIND env variable.
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
