10000 v1 QA · Issue #2100 · cometbft/cometbft · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

v1 QA #2100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
7 of 10 tasks
Tracked by #578
melekes opened this issue Jan 23, 2024 · 1 comment
Closed
7 of 10 tasks
Tracked by #578

v1 QA #2100

melekes opened this issue Jan 23, 2024 · 1 comment
Assignees
@mzabaluev @melekes @jmalicevic @hvanz
Labels
major-priority A major, long-running priority for the team qa Quality assurance tracking A complex issue broken down into sub-problems
Milestone
2024-Q2

Comments

@melekes
Copy link
Contributor
melekes commented Jan 23, 2024 •
edited by adizere
Loading

The focus of this QA process for releasing v1 is to:

  • test regressions with respect to our baseline, the v0.38 results (see #31),
  • assess the performance of new features, including PBTS and storage optimizations,
  • perform a stress test,
  • optionally, test experimental features for bandwidth improvement.

The main goal of the QA process is to validate that there are no meaningful, substantial regression from the previous version. There is a regression if we see that there is a difference bigger than ~10% in the results.

Preliminary QA

### Tasks
- [ ] https://github.com/cometbft/cometbft/issues/903
- [ ] https://github.com/cometbft/cometbft/issues/2058
- [ ] ~~stress test with high volumes of txs / big txs (aka "p2p storm")~~
- [ ] ~~update default parameters based on ^ (_plus other possible fixes tbd_) e.g. mempool parameters~~
- [x] Run preliminary test of PBTS branch on 200 nodes
- [ ] https://github.com/cometbft/cometbft/issues/2323
- [ ] https://github.com/cometbft/cometbft/issues/2460

Hypotheses to verify

  • new key layout helps access time and compaction
  • bandwidth optimizations reduced the amount of data sent/received
  • with updated default parameters, CometBFT is resilient against high volumes of txs / big txs.

Questions

  1. Do we want to make some bandwidth improvements non-experimental?

Improve end-to-end test framework

New metrics

Final QA

Each task will update a section on the final QA report.

### Tasks
- [ ] ~~Capture metrics of v0.38 baseline (Optional, to do only if existing v0.38 metrics can't be reused)~~
- [ ] #2461
- [ ] #2513
- [ ] https://github.com/cometbft/cometbft/issues/2656
- [ ] https://github.com/cometbft/cometbft/issues/2657
- [ ] ~~Test storage on experiments longer than 90s~~
- [ ] https://github.com/cometbft/cometbft/issues/2746
- [x] Improve QA instructions as the tests are run #2405
@melekes melekes added major-priority A major, long-running priority for the team tracking A complex issue broken down into sub-problems labels Jan 23, 2024
@melekes melekes self-assigned this Jan 23, 2024
@melekes melekes added this to CometBFT Jan 23, 2024
@github-project-automation github-project-automation bot moved this to Todo in CometBFT Jan 23, 2024
@melekes melekes mentioned this issue Jan 23, 2024
Closed
31 tasks
@melekes melekes moved this from Todo to In Progress in CometBFT Jan 23, 2024
@melekes melekes added this to the 2024-Q1 milestone Jan 23, 2024
@hvanz hvanz added the qa Quality assurance label Jan 24, 2024
@faddat

This comment was marked as off-topic.

Sign in to view
@adizere adizere modified the milestones: 2024-Q1, 2024-Q2 Apr 1, 2024
@sergio-mena sergio-mena mentioned this issue Apr 8, 2024
Merged
@adizere adizere closed this as completed Apr 15, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in CometBFT Apr 15, 2024
@melekes melekes mentioned this issue Jun 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mzabaluev mzabaluev

@melekes melekes

@jmalicevic jmalicevic

@hvanz hvanz

Labels
major-priority A major, long-running priority for the team qa Quality assurance tracking A complex issue broken down into sub-problems
Projects
No open projects
CometBFT
Status: Done
Milestone
2024-Q2
Development

No branches or pull requests
6 participants
@mzabaluev @melekes @adizere @faddat @jmalicevic @hvanz
0