Set of latency benchmarks testing round trip time (RTT) between threads or processes via FIFO data structures and messaging systems.
To run the Java benchmarks execute the Gradle script in the base directory.
$ ./gradlew
$ ./gradlew runJavaBenchmarks
or just the Aeron benchmarks
$ ./gradlew runAeronJavaBenchmarks
To generate the benchmarks, execute the cppbuild script from the base directory.
$ cppbuild/cppbuild
To run the benchmarks, execute the individual benchmarks.
$ cppbuild/Release/binaries/baseline
$ cppbuild/Release/binaries/aeronExclusiveIpcBenchmark
$ cppbuild/Release/binaries/aeronIpcBenchmark
$ cppbuild/Release/binaries/aeronExclusiveIpcNanomark
$ cppbuild/Release/binaries/aeronIpcNanomark
Note: On MacOS, it will be necessary to set DYLD_LIBRARY_PATH for the Aeron
driver shared library. For example:
$ env DYLD_LIBRARY_PATH=cppbuild/Release/aeron-prefix/src/aeron-build/lib cppbuild/Release/binaries/aeronIpcBenchmark
The binaries with Benchmark in the name use Google Benchmark and only displays average times.
While the binaries with Nanomark in the name use Nanomark (included in the source) and displays full histograms.
To pick a specific tag for Aeron, specify --aeron-git-tag parameter when invoking cppbuild script.
For example:
cppbuild/cppbuild --aeron-git-tag="1.27.0"will use Aeron 1.27.0 release.
The scripts directory contains scripts to run the remote benchmarks, i.e. the benchmarks that test sending data
between the client (benchmarking harness) and the server in different scenarios.
uk.co.real_logic.benchmarks.remote.LoadTestRig implements the benchmarking harness. The
uk.co.real_logic.benchmarks.remote.Configuration class provides the configuration for the benchmarking harness.
The common way to run the benchmarks is to use benchmark-runner to run the benchmarking harness with the given client.
This script will run benchmarks multiple times for each permutation of the configuration parameters.
For example:
$ ./benchmark-runner --output-file "echo-test" --messages "1000, 5000" --burst-size "1, 10" --message-length "32, 224, 1376" "aeron/echo-client"Will execute aeron/echo-client script for every permutation of the number of messages, burst size and the message
length. Each execution will run five times doing ten measurement iterations.
The first execution will be with 1000 messages, burst size of 1 and message payload length of 32 bytes. The
second will have a different message length (i.e. 224 bytes) etc.
To aggregate the results of the multiple runs into a single file there is the aggregate-results script.
For example if the results directory contains the following files:
results
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-0.hdr
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-1.hdr
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-2.hdr
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-3.hdr
└── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-4.hdrRunning:
./aggregate-results resultsWill produce the following result:
results
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-0.hdr
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-1.hdr
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-2.hdr
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-3.hdr
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-4.hdr
├── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-combined.hdr
└── echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-report.hgrmwhere echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-combined.hdr is the
aggregated histogram of five runs and the
echo-test_1000_1_32_c7a083c84b45f77fdee5cedc272d898d44b6e18deaf963b3e2b2c074006b0b10-report.hgrm is an export of the
aggregated histogram that can be plotted using http://hdrhistogram.github.io/HdrHistogram/plotFiles.html.
This section lists the systems under test which implement remote benchmarks.
Aeron remote benchmarks implement the following test scenarios:
-
Echo benchmark (aka ping-pong)
-
Live replay from remote archive
The client publishes messages to the server using publication over UDP. The server pipes those messages into a local IPC publication which records them into an archive. Finally, the client subscribes to the replay from that archive over UDP.
- Live recording, i.e. client runs records a publication into local archive
The client publishes messages over UDP to the server. It also has a recording running on that publication using local archive. The server simply pipes message back. Finally, the client performs a controlled poll on the subscription from the server limited by the "recording progress" which it gets via the recording events.
The biggest difference between scenario 2 and this scenario is that there is no replay of recorded message and hence no reading from the disc of the saved data but still allowing consumption of those messages that were successfully persisted.
Please the documentation in the aeron directory for more information.
For gRPC there is only ping-pong test with two different implementations:
- Blocking client - client uses blocking API to send messages.
- Streaming client - client uses streaming API to send/receive messages.
Please the documentation in the grpc directory for more information.
Unlike gRPC that simply echoes messages Kafka will persist them so that the test is similar to the Aeron's replay from the remote archive.
Please the documentation in the kafka directory for more information.
Copyright 2015-2022 Real Logic Limited.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.