Benchi is a minimal benchmarking framework designed to help you measure the performance of your applications and infrastructure. It leverages Docker to create isolated environments for running benchmarks and collecting metrics.
It was developed to simplify the process of setting up and running benchmarks for Conduit.
- Docker Integration: Define and manage your benchmarking environments using Docker Compose.
- Metrics Collection: Collect and export metrics in CSV format for further analysis.
- Custom Hooks: Define custom hooks to run commands at various stages of the benchmark.
- Progress Monitoring: Real-time monitoring of container statuses and metrics during the benchmark run.
To install Benchi, download the latest release or install it using Go:
go install github.com/conduitio/benchi/cmd/benchi@latestAlternatively, you can install just the Benchi binary on a Linux OS with:
curl https://raw.githubusercontent.com/ConduitIO/benchi/main/install.sh | shRun benchi and point -config to a benchmark configuration file.
The repository includes an example benchmark, which can be run
using the following command:
benchi -config ./example/bench-kafka-kafka/bench.ymlRunning the benchmark will store the results in a folder named after the current
date and time inside of results (e.g. results/20060102_150405). You can
adjust the output folder using the -out flag.
The output folder will contain two files:
benchi.log: Log file containing the full output of benchi.aggregated-results.csv: Aggregated metric results from all collectors and all tests. The results are aggregated using a trimmed mean, where the top and bottom 5% of the results are removed. Benchi also disregards any 0 values from the start and end of the test, to accomodate for warm-up and cool-down periods.
The output folder will also contain one folder per benchmark run (i.e. per test and tool combination). Each benchmark run folder will contain:
infra_NAME.log: Log file containing the output of the infrastructure docker containers, split per infra service.tool_NAME.log: Log file containing the output of the tool docker containers.COLLECTOR.csv: Raw metrics collected using the corresponding metrics collector.
-config: Path to the benchmark config file (required).-out: Path to the output folder (default:./results/${now}*).-tool: Filter tool to be tested (can be provided multiple times).-tests: Filter test to run (can be provided multiple times).
* ${now} is replaced with the current time formatted as YYYYMMDD_HHMMSS.
Benchi creates a Docker network named benchi to connect the infrastructure
services and tools. This network is created automatically and removed after the
benchmark run. Please make sure to connect your services to this network to
ensure they can communicate with each other.
Example Docker Compose configuration:
services:
my-service:
networks:
- benchi
networks:
benchi:
external: trueBenchi runs all Docker Compose commands using the same environment variables as the current shell. This means that you can use environment variables to pass values to your services.
For instance, having the following Docker Compose configuration:
services:
my-service:
environment:
- MY_ENV_VAR=${MY_ENV_VAR}You can inject the environment variable by running Benchi as follows:
MY_ENV_VAR=my-value benchi -config ./my-benchmark.ymlBenchi uses a YAML configuration file to define the benchmark in combination with Docker Compose configurations.
Below is an example configuration:
infrastructure:
database:
compose: "./compose-database.yml"
cache:
compose: "./compose-cache.yml"
tools:
my-app:
compose: "./compose-my-app.yml"
metrics:
prometheus:
collector: "prometheus"
settings:
url: "http://localhost:9090/metrics"
queries:
- name: "http_requests_rate"
query: "rate(request_count{endpoint=hello}[2s])"
unit: "req/s"
interval: "1s"
tests:
- name: Endpoint Load
duration: 2m
steps:
pre-infrastructure:
post-infrastructure:
- name: Setup Database
container: database
run: /scripts/setup-database.sh
pre-tools:
post-tools:
pre-test:
during:
- name: Run Load Test
container: my-app
run: /scripts/run-load-test.sh
post-test:
pre-cleanup:
- name: Cleanup
container: my-app
run: /scripts/cleanup.sh
post-cleanup:The infrastructure section defines the Docker Compose configurations for the
infrastructure services required for the benchmark. Each service is identified
by a custom name, used in logging and to correlate overridden configurations
specified in a test (see tests). The path to the docker compose file
is relative to the location of the benchmark configuration file.
Example:
infrastructure:
name-of-infrastructure-service:
compose: "./path/to/compose-file.yml"The tools section defines the Docker Compose configurations for the tools
being benchmarked. Each tool is identified by a custom name, used in logging and
to correlate overridden configurations specified in a test (see tests.
The path to the docker compose file is relative to the location of the benchmark
configuration file.
Example:
tools:
name-of-tool:
compose: "./path/to/compose-file.yml"The metrics section defines the metric collectors running during the
benchmark. Each metric collector has a custom name used for logging. The
collector field specifies the type of metric collector to use. The settings
field contains the configuration for the chosen collector.
Example:
metrics:
name-of-metric-collector:
collector: "conduit"
settings:
url: "http://localhost:8080/metrics"Note
Metrics collectors run in the benchi process, which runs outside of docker on the host machine. Ensure that the metric collector can access the endpoints of the services being benchmarked by exposing the necessary ports in the Docker Compose configurations.
See collectors for available collectors and their configurations.
The tests section defines the benchmarks to run. Each test has a custom name
used for logging. The duration field specifies the duration of the test. The
steps field contains the commands to run at various stages of the benchmark.
The steps field contains the following stages:
pre-infrastructure: Commands to run before starting the infrastructure services.post-infrastructure: Commands to run after starting the infrastructure services.pre-tools: Commands to run before starting the tools.post-tools: Commands to run after starting the tools.pre-test: Commands to run before starting the test.during: Commands to run during the test.post-test: Commands to run after the test.pre-cleanup: Commands to run before cleaning up the test.post-cleanup: Commands to run after cleaning up the test.
Note
Steps are generally executed sequentially and in the order specified in the
configuration. However, the during step is an exception, as all commands
under this step are executed concurrently and will run for the duration of the
test.
Each hook can run its commands either in an existing container or in a temporary
container created from a specified image. The container field specifies the
name of the container to run the commands in. The image field specifies the
image to use for the temporary container. If neither container nor image is
specified, the commands will run in a temporary container using the
alpine:latest image. Note that running a custom script in a container requires
the container to contain the necessary script (hint: mount the script as a
volume).
You can optionally configure the tools field to run a hook only for certain
tools. If the field is not present or empty, the hook is applied for all tools.
Example:
tests:
- name: My Test
duration: 2m
steps:
pre-infrastructure:
post-infrastructure:
# This script will run after the infrastructure services have started.
# It will run in the `database` container (presumably started by the
# infrastructure service).
- name: Setup Database
container: database
run: /scripts/setup-database.sh
pre-tools:
post-tools:
pre-test:
during:
# This script will run for the duration of the test (2 minutes). It will
# run in the `my-app` container (presumably started by the tool). The
# script will receive a SIGINT signal when the test duration is over.
- name: Run Load Test
container: my-app
tools:
- my-app # Only run this hook when benchmarking my-app
run: /scripts/run-load-test.sh
post-test:
pre-cleanup:
# This script will run before infrastructure and tools containers are
# stopped and removed. It will run in a temporary container created
# from the `busybox:latest` image and connected to the `benchi` network.
- name: Cleanup
image: "busybox:latest"
run: |
echo "Cleaning up..."
sleep 5
post-cleanup:You can also include custom infrastructure and tools configurations to
override the default configurations specified in the infrastructure and
tools sections. Note that the global configurations will still be applied, the
additional configurations are merged with the global configurations (see
merging compose files).
This can be useful to inject custom configurations for a specific test.
Important
Since the provided configurations are merged with the global configurations, any paths specified in the custom docker compose configurations should be relative to the location of the global docker compose configuration. See merging rules.
Example:
tests:
- name: My Test
duration: 2m
infrastructure:
name-of-infrastructure-service:
compose: "./compose-file-infra.override.yml"
tools:
name-of-tool:
compose: "./compose-file-tool.override.yml"Collectors are used to collect metrics from various sources during the benchmark run. The collected metrics are exported in CSV format for further analysis.
Collectors are configured in the metrics section of the benchmark
configuration file.
Supported collectors:
The Conduit metrics collector tracks the throughput for each configured pipeline in Conduit.
Settings:
url: URL of the Conduit metrics endpoint (needs to be reachable from the benchi process).pipelines: Array of pipelines to track.
metrics:
my-conduit-collector:
collector: "conduit"
settings:
url: "http://localhost:8080/metrics"
pipelines:
- "pipeline1"
- "pipeline2"Metrics:
msg-rate-per-second[PIPELINE]: Messages per second per pipeline.msg-megabytes-in-per-second[PIPELINE]: Incoming megabytes per second per pipeline (measured as records read by the source connector).msg-megabytes-out-per-second[PIPELINE]: Outgoing megabytes per second per pipeline (measured as records written by the destination connector).
The Docker metrics collector tracks the container resource usage during the benchmark run.
Settings:
containers: Array of containers to track.
metrics:
my-docker-collector:
collector: "docker"
settings:
containers:
- "my-app"
- "database"
- "cache"Metrics:
cpu-percentage[CONTAINER]: CPU usage in percent at a specific point in time per container.memory-usage[CONTAINER]: Memory usage in megabytes at a specific point in time per container.
The Kafka metrics collector tracks the throughput for each configured topic in Apache Kafka.
The collector expects Kafka to expose a Prometheus metrics endpoint via the Prometheus JMX exporter. To see how to configure it, check out the example configuration.
Settings:
url: URL of the Kafka metrics endpoint (needs to be reachable from the benchi process).topics: Array of topics to track.
metrics:
my-kafka-collector:
collector: "kafka"
settings:
url: "http://localhost:7071/metrics"
topics:
- "topic1"
- "topic2"Metrics:
msg-rate-in-per-second[TOPIC]: Incoming messages per second per topic.msg-megabytes-in-per-second[TOPIC]: Incoming megabytes per second per topic.msg-megabytes-out-per-second[TOPIC]: Outgoing megabytes per second per topic.
The Prometheus metrics collector continuously scrapes a metrics endpoint, stores the metrics in memory and queries them using PromQL.
It is expected that the query returns a matrix with a single series. The query should be a ranged query, which will be evaluated between the start of the test and the end of the test.
Settings:
url: URL of the Prometheus metrics endpoint.queries: Array of queries to run.name: Name of the query.query: PromQL query.unit: Unit of the query (optional, only for displaying in the CLI).interval: Resolution of the ranged query.
metrics:
my-prometheus-collector:
collector: "prometheus"
settings:
url: "http://localhost:8080/metrics"
queries:
- name: "http_request_success_rate"
query: "rate(request_count{endpoint=hello,status=200}[2s])"
unit: "req/s"
interval: "1s"
- name: "http_request_fail_rate"
query: "rate(request_count{endpoint=hello,status!=200}[2s])"
unit: "req/s"
interval: "1s"Metrics are user defined using the queries field. The name field is used as
the metric name in the CSV output.
- Benchi fails with the error
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
Benchi is communicating with Docker using the default Docker socket. If you are using Docker Desktop, you can enable it under Settings -> Advanced and check the box for "Allow the default Docker socket to be used".
Benchi is licensed under the Apache License, Version 2.0. See the
LICENSE file for more details.