8000 [Tracking issue] flaky tests · Issue #1923 · cometbft/cometbft · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[Tracking issue] flaky tests #1923

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
4 of 24 tasks
faddat opened this issue Jan 2, 2024 · 2 comments
Open
4 of 24 tasks

[Tracking issue] flaky tests #1923

faddat opened this issue Jan 2, 2024 · 2 comments
Labels
bug Something isn't working non-deterministic An issue that is difficult to reproduce because it is non-deterministic testing related to unit testing in general tracking A complex issue broken down into sub-problems

Comments

@faddat
Copy link
Contributor
faddat commented Jan 2, 2024 •
edited by cason
Loading

Bug Report

We have several Go test units that fail randomly, are flaky, in particular when running by GitHub (e.g., in pull requests).

The goal of this issue is to keep track of them and define what to with the associated test units. The possibilities include:

  1. Remove the test unit, justifying the removal through a PR and associated issue. This approach could be taken with test units that are particular hard to fix or understand and whose removal would presumably not affect the correctness of the code, upon the introduction of changes
  2. Fix the test unit. This is the optimal solution but it takes time, in some cases a lot of time. Notice that at the moment this text is written (Q3-2024), we don't have in our priority list fixing flaky tests.
  3. Add comments to the code to inform that a particular test unit is flaky, possibly linking the corresponding issue. This does not solve the problem but helps onboarding new collaborators.

Main branch

List of potential flaky tests

List produced by @faddat in January 2024. Links for the failed runs were removed because not available any longer.
@faddat faddat added bug Something isn't working needs-triage This issue/PR has not yet been triaged by the team. labels Jan 2, 2024
@faddat faddat changed the title Multiple flaky tests Flaky Test Tracking Issue Jan 2, 2024
@faddat
Copy link
Contributor Author
faddat commented Jan 2, 2024 •
edited
Loading

I've begun to submit fixes to the flaky tests. I would like to urge the maintainers to check the fixes extremely harshly, such that none of the fixes endangers the purpose of any of the tests.

For example, you might try merging all of the fixes to one branch, running ci hundreds of times, and seeing if there are failures.

But I would also recommend that for each of the sumbitted PR's the underlying code be examined just as harshly as its test.

@melekes melekes added non-deterministic An issue that is difficult to reproduce because it is non-deterministic and removed needs-triage This issue/PR has not yet been triaged by the team. labels Jan 9, 2024
@faddat
Copy link
Contributor Author
faddat commented Jan 19, 2024 •
edited
Loading

I think I figured out TestReader

--- FAIL: TestReader (0.44s)
    io_test.go:94: r.Status(1)
        expected: {2024-01-19 18:27:55.48 +0000 UTC 10 1 100 100 100 100 0 100ms 0s 0s 0.000% true}
        got     : {2024-01-19 18:27:55.48 +0000 UTC 10 1 100 100 100 100 0 100ms 20ms 0s 0.000% true}
FAIL

So basically, we're expecting that to do.... something in 0s but it takes 20ms. Also, I kinda think we can tolerate the 20ms.

cometbft/internal/flowrate/io_test.go

Lines 82 to 83 in d766d20

// Active, Bytes, Samples, InstRate, CurRate, AvgRate, PeakRate, BytesRem, Start, Duration, Idle, TimeRem, Progress
want := []Status{

But maybe we cannot? Reviewing this, this looks ... time related? This is looking for idle to always be 0s.

This was referenced Jan 19, 2024
Closed
Closed
This was referenced Feb 10, 2024
Open
Merged
Closed
@cason cason added the tracking A complex issue broken down into sub-problems label Jun 11, 2024
@cason cason changed the title Flaky Test Tracking Issue [Tracking issue] flaky tests Jun 11, 2024
@cason cason added the testing related to unit testing in general label Jul 29, 2024
@cason cason added this to CometBFT Jul 29, 2024
@github-project-automation github-project-automation bot moved this to Todo in CometBFT Jul 29, 2024
@melekes melekes mentioned this issue Sep 2, 2024
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working non-deterministic An issue that is difficult to reproduce because it is non-deterministic testing related to unit testing in general tracking A complex issue broken down into sub-problems
Projects
No open projects
CometBFT
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cason @melekes @faddat
0