8000 QA: v0.34 - 1/3 TM 2/3 CMT by lasarojc · Pull Request #297 · cometbft/cometbft · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

QA: v0.34 - 1/3 TM 2/3 CMT #297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Feb 15, 2023
Merged

QA: v0.34 - 1/3 TM 2/3 CMT #297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
6f204f4
adding the 2/3 comet 1/3 tm case, without explanations
lasarojc Feb 9, 2023
20d157a
moving experiment to folder to standardize names
lasarojc Feb 10, 2023
36c3c9a
Description of how to run the prometheus plotter file
lasarojc Feb 10, 2023
04e3281
regenerating figures and deleting the old ones
lasarojc Feb 10, 2023
1910c18
Merge branch 'main' into lasarojc/0.34-qa-tm1cmt2
lasarojc Feb 10, 2023
11c19f3
moving the reporting scripts to this repo
lasarojc Feb 10, 2023
051cc1a
Updating CMT1/3 TM2/3 latency figures
lasarojc Feb 10, 2023
0239330
Updating prometheus metrics for cmt1 tm2 case
lasarojc Feb 10, 2023
b43f316
Updating scripts
lasarojc Feb 10, 2023
9949483
extending the window to capture the long tail of experiments
lasarojc Feb 10, 2023
f32e764
improved plotter, let's you include the average in the graph
lasarojc Feb 10, 2023
77534bc
Updating prometheus metrics for cmt1 tm2 case
lasarojc Feb 10, 2023
8000
e442937
Revert to referencing TM where it makes sense
lasarojc Feb 10, 2023
6e9b0fc
Updating prometheus metrics for cmt2 tm1 case
lasarojc Feb 10, 2023
14554df
Updating prometheus metrics for cmt2 tm1 case and references in the R…
lasarojc Feb 10, 2023
3ef785c
Updating prometheus metrics for cmt1 tm2 case
lasarojc Feb 10, 2023
8ad74de
Fixing section levels. Changed results to table
lasarojc Feb 10, 2023
5bebaf5
reducing time window to 20s
lasarojc Feb 10, 2023
ac1b01a
reducing time window to 20s
lasarojc Feb 10, 2023
d0e3666
Start splitting the files
lasarojc Feb 10, 2023
c3cf8a1
reducing time window to 20s in the script
lasarojc Feb 10, 2023
02ab792
Merge branch 'main' into lasarojc/0.34-qa-tm1cmt2
lasarojc Feb 11, 2023
5ca8b61
Update the data and comments for homogeneous network (#312)
sergio-mena Feb 11, 2023
2659242
making file names consitent with script output and across sections. R…
lasarojc Feb 13, 2023
805143c
Merge branch 'main' into lasarojc/0.34-qa-tm1cmt2
lasarojc Feb 13, 2023
f4336d1
Plot to the imgs folder. Don't block between plots.
lasarojc Feb 14, 2023
4423754
Adding report for half-half configuration. Update scripts include the…
lasarojc Feb 14, 2023
ecaf727
Fix typo in the `requirements.txt` file.
lasarojc Feb 15, 2023
142231a
Moving instructions to run to the reporting
lasarojc Feb 15, 2023
89b6885
Small tweaks in the text
lasarojc Feb 15, 2023
8b7f477
Merge branch 'main' into lasarojc/0.34-qa-tm1cmt2
lasarojc Feb 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/qa/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ at the time the Quality Assurance process was applied for that release.
File [method](./method.md) keeps track of the process followed to obtain the results
used to decide if a release is passing the Quality Assurance process.
The results obtained in each release are stored in their own directory.
The following releases have undergone the Quality Assurance process:
The following releases have undergone the Quality Assurance process, and the corresponding reports include detailed information on tests and comparison with the baseline.

* [v0.34.x](./v034/), which was tested just before releasing v0.34.22
* [v0.37.x](./v037/), with v.34.x acting as a baseline
* [TM v0.34.x](./v034/TMCore.md) - Tested prior to releasing Tendermint Core v0.34.22.
* [v0.34.x](./v034/CometBFT.md) - Tested prior to releasing v0.34.27, using TM v0.34.x results as baseline.
* [v0.37.x](./v037/) - with TM v.34.x acting as a baseline
13 changes: 11 additions & 2 deletions docs/qa/method.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,15 +192,24 @@ The CometBFT team should improve it at every iteration to increase the amount of
in step 2, follow the instructions for the [`latency_throughput.py`] script.
This plot is useful to visualize the saturation point.

[`latency_throughput.py`]: ../../scripts/qa/reporting/README.md
[`latency_throughput.py`]: ../../scripts/qa/reporting/README.md#Latency-vs-Throughput-Plotting

14. Alternatively, follow the instructions for the [`latency_plotter.py`] script.
This script generates a series of plots per experiment and configuration that my
help with visualizing Latency vs Throughput variation.

[`latency_plotter.py`]: ../../scripts/qa/reporting/README.md#Latency-vs-Throughput-Plotting-version-2

#### Extracting Prometheus Metrics

1. Stop the prometheus server if it is running as a service (e.g. a `systemd` unit).
2. Unzip the prometheus database retrieved from the testnet, and move it to replace the
local prometheus database.
3. Start the prometheus server and make sure no error logs appear at start up.
4. Introduce the metrics you want to gather or plot.
4. Identify the time window you want to plot in your graphs.
5. Execute the [`prometheus_plotter.py`] script for the time window.

[`prometheus_plotter.py`]: ../../scripts/qa/reporting/README.md#prometheus-metrics

## Rotating Node Testnet

Expand Down
Loading
0