8000 connection_checker_test.js has flaky tests · Issue #8688 · google/blockly · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

connection_checker_test.js has flaky tests #8688

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
1 task done
BenHenning opened this issue Dec 5, 2024 · 4 comments · Fixed by #8754
Closed
1 task done

connection_checker_test.js has flaky tests #8688

BenHenning opened this issue Dec 5, 2024 · 4 comments · Fixed by #8754
Labels
issue: bug Describes why the code or behaviour is wrong

Comments

@BenHenning
Copy link
Contributor
BenHenning commented Dec 5, 2024 •
edited
Loading

Check for duplicates

  • I have searched for similar issues before opening a new one.

Description

Sometimes when running npm run test locally, I see failures in connection_checker_test.js. I found this while investigating #8676 and it occurs to me that it could be environment specific.

The specific tests failing are:

  • Connection checker Dragging Checks Stacks Do not splice into unmovable stack (about 3% flaky): test link.
  • Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack (about 14% flaky): test link.

Reproduction steps

  1. Run tests locally several times.
  2. Observe that two specific tests seem to fail at a fairly high rate.

Some techniques that I find helpful:

  • Run the tests in headless mode (seems to be better performance, and I'm developing over a remote shell): CI=true npm run test.
  • Comment out all but mocha from tasks in scripts/gulpfiles/test_tasks.js (so that only Mocha tests run).
  • Replace the top-level suite call with suite.only in tests/mocha/connection_checker_test.js (so that only this suite runs).
  • Use one of the scripts in https://gist.github.com/BenHenning/a4daa87d7983026411ac12f3b357005b to run the tests multiple times to detect flakes (I prefer the Kotlin one since it handles output much better).

Stack trace

Output from running run_blockly_tests.main.kts per the Gist above:

Running Blockly tests in /usr/.../blockly 100 time(s) across X core(s):
Completed 10/100 (finished: 13.386s, avg: 1.3385999s, rem: ~120.474s)
Completed 20/100 (finished: 16.146s, avg: 1.4765999s, rem: ~118.128s)
Completed 30/100 (finished: 19.628s, avg: 1.6386666s, rem: ~114.706665s)
Completed 40/100 (finished: 20.929s, avg: 1.7522249s, rem: ~105.1335s)
Completed 50/100 (finished: 22.803s, avg: 1.85784s, rem: ~92.892s)
Completed 60/100 (finished: 25.131s, avg: 1.9670501s, rem: ~78.682s)
Completed 70/100 (finished: 27.237s, avg: 2.0751429s, rem: ~62.254284s)
Completed 80/100 (finished: 29.679s, avg: 2.1867375s, rem: ~43.73475s)
Completed 90/100 (finished: 31.924s, avg: 2.298478s, rem: ~22.984777s)
Completed 100/100 (finished: 22.88s, avg: 2.29743s, rem: ~0.0s)

Finished running tests in 34.086 seconds (total execution time: 229.743 seconds). Results:
83/100 runs passed.
17/100 runs failed.
Specific failures:
- 3 times: Connection checker Dragging Checks Stacks Do not splice into unmovable stack
- 14 times: Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack

The failure for the 'Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack' test was:

"Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack" failed: Should connect below an unmovable stack block: expected false to be true

I didn't collect the stack trace of the other test failure.

Screenshots

N/A since this is a test assertion flake.

Browsers

No response
@BenHenning BenHenning added issue: bug Describes why the code or behaviour is wrong issue: triage Issues awaiting triage by a Blockly team member labels Dec 5, 2024
@BenHenning
Copy link
Contributor Author

Note that I just ran the entire Mocha test suite 100 times and these two tests were the only ones flaky for me (the most recent run showed the 'do not splice' test as 8% flaky and the 'connect to bottom of unmovable stack' test as 11% flaky).

I also did a run of just those two flaky tests in isolation over 1000 runs to get a better estimate of flakiness. Over 1000 runs:

  • 9.3% flaky: Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack
  • 9.2% flaky: Connection checker Dragging Checks Stacks Do not splice into unmovable stack

@BenHenning BenHenning mentioned this issue Dec 6, 2024
Merged
1 task
@BenHenning
Copy link
Contributor Author

I'm not sure who has been affected by these flakes in CI, but I just ran into one of them in #8689.

@cpcallen cpcallen removed the issue: triage Issues awaiting triage by a Blockly team member label Dec 6, 2024
@BenHenning
Copy link
Contributor Author
BenHenning commented Jan 22, 2025 •
edited
Loading

#8743 ran into this again today. Still unclear as to the cause.

@RoboErikG
Copy link
Contributor
RoboErikG commented Jan 24, 2025 81AB

I was able to reproduce this locally with some additional logging. It looks like the sorting of getAllBlocks is inconsistent, which leads to the wrong blocks being assigned during the suite setup. Here's an example log where I printed out the blocks being used for the test along with what Block they were expected to be:

"Connection checker Dragging Checks Stacks Connect to bottom of unmovable stack" failed: Should connect below an unmovable stack block
Block B: <block xmlns="https://developers.google.com/blockly/xml" type="text_print" id="A" x="64" y="34"><next><block type="text_print" id="B" movable="false"></block></next></block>

Block C: <block xmlns="https://developers.google.com/blockly/xml" type="text_print" id="B" movable="false" x="64" y="61"></block>: expected false to be true

@gonfunko gonfunko mentioned this issue Jan 30, 2025
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
issue: bug Describes why the code or behaviour is wrong
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Fix flaky connection checker test. gonfunko/blockly
3 participants
@cpcallen @RoboErikG @BenHenning
0