8000 test: tiktoken test timing assertions are unreliable in CI · Issue #3103 · RooVetGit/Roo-Code · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

test: tiktoken test timing assertions are unreliable in CI #3103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
KJ7LNW opened this issue May 2, 2025 · 3 comments
Closed

test: tiktoken test timing assertions are unreliable in CI #3103

KJ7LNW opened this issue May 2, 2025 · 3 comments
Labels
bug Something isn't working

Comments

@KJ7LNW
Copy link
Collaborator
KJ7LNW commented May 2, 2025

The tiktoken test is failing in CI due to timing assertions that cannot be reliably tested in a CI environment.

expect(received).toBeLessThanOrEqual(expected)

Expected: <= 0.3159959999975399
Received:    10.159213999999338

Timing-based tests are inherently flaky in CI environments due to varying system load, resources, and other factors outside our control.

The specific test failing is in src/utils/__tests__/tiktoken.test.ts line 140:

expect(duration2).toBeLessThanOrEqual(duration1 * 1.5)

@cte This test was recently added and is causing false positives in unrelated pull requests. What could be done to handle timing assertions in CI environments?
@dosubot DosuBot
Copy link
dosubot bot commented May 2, 2025

Timing assertions in CI environments can be challenging due to variability in system performance. Here are some strategies to handle these issues:

  1. Increase Tolerance: Adjust the expected timing threshold to account for variability. This can reduce false positives by allowing more leeway in timing differences.

  2. Mocking and Stubbing: Replace real-time measurements with mocked or stubbed values that simulate expected behavior without relying on actual timing.

  3. Remove Timing Assertions: If timing is not critical to the functionality being tested, consider removing these assertions altogether.

  4. Environment-Specific Tests: Run timing-sensitive tests in a controlled environment separate from CI, where system load and resources can be managed.

  5. Use Performance Profiling Tools: Integrate tools that can provide more consistent performance metrics, which might be less susceptible to CI variability.

  6. Conditional Testing: Implement conditional logic to skip timing tests in CI environments, running them only in local or dedicated performance testing setups.

These approaches can help mitigate the impact of timing variability in CI environments and reduce false positives in your tests.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot dosubot bot added the bug Something isn't working label May 2, 2025
@cte
Copy link
Collaborator
cte commented May 2, 2025

We can remove it for now - Roo added that test and I didn't look closely.

@mrubens
Copy link
Collaborator
mrubens commented May 4, 2025

Test has been removed

@mrubens mrubens closed this as completed May 4, 2025
@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap May 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Roo Code Roadmap
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mrubens @cte @KJ7LNW
0