8000 New GNU tests failures for 'tests/misc/chcon-fail' and 'tests/misc/false-status' · Issue #3025 · uutils/coreutils · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

New GNU tests failures for 'tests/misc/chcon-fail' and 'tests/misc/false-status' #3025

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
rivy opened this issue Feb 2, 2022 · 3 comments
Closed

New GNU tests failures for 'tests/misc/chcon-fail' and 'tests/misc/false-status' #3025

rivy opened this issue Feb 2, 2022 · 3 comments

Comments

@rivy
Copy link
Member
rivy commented Feb 2, 2022

Looks like this error has started showing up in 'main' merges and new PRs...

GNU test failed: tests/misc/chcon-fail. tests/misc/chcon-fail is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/misc/false-status. tests/misc/false-status is passing on 'main'. Maybe you have to rebase?

In going back over the tests for the merges, I don't see where it slipped into the commit chain.
The first time I see this particular error group is in a764fdf, but that commit only changed the README, so it can't be the source of the issue.

I'm re-running the job to see if it's just a flakiness issue, but I'm seeing it in multiple commits and PRs now.

@sylvestre, I think you wrote the GNU tests CI script, yes? Maybe you can see where the problem is?
@rivy
Copy link
Member Author
rivy commented Feb 2, 2022 •
edited
Loading

I ran a manual test of the comparison logic "Compare failing tests against main" between the first failing commit run to 'main' and the run for the immediately prior (successful) 'main' commit.

My comparison (using the logic from "Compare failing tests against main") showed no differences at all... no new success or new failure. The job run, however, shows...

GNU test failed: tests/misc/chcon-fail. tests/misc/chcon-fail is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/misc/false-status. tests/misc/false-status is passing on 'main'. Maybe you have to rebase?
...
Congrats! The gnu test tests/dd/stderr is now passing!

I think I must be comparing against a baseline log which differs from what the CI is using, but I'm not sure how to ID which log CI is using for comparison.

Maybe we should create/upload a hash ID of the file set as an artifact as well to discriminate what is being compared? Something like:

FILESET_HASH=$(LC_ALL=C tar cf - gnu-result.json test-suite.log **/*.log --sort=name | sha1sum | cut --delim=" " -f 1)
touch "${FILESET_HASH}.sha1"

and then upload that as a separate artifact and display it in the job logs.

@tertsdiepraam tertsdiepraam mentioned this issue Feb 3, 2022
Merged
@rivy
Copy link
Member Author
rivy commented Feb 4, 2022

@sylvestre , @tertsdiepraam , I'm 90% through a rewrite of the GnuTests CI workflow which adds some additional mechanisms and logging to identify where errors are occurring and to be more flexible if a baseline reference test suite is unavailable (ie, never done [for new forks] or expired). I'll post a PR likely later today or tomorrow.

Hopefully, with that PR we can track down/fix this issue. 🤞🏻

@rivy
Copy link
Member Author
rivy commented Feb 5, 2022

PR #3077 for revision of GnuTests with additional logging and flexibility for reference comparison is cooking here in CI.

It might fix this issue when once merged.
At least, it should be able to help track down what the problem is...

@rivy rivy mentioned this issue Feb 5, 2022
Merged
@rivy rivy closed this as completed Apr 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
3405 No milestone
Development

No branches or pull requests
1 participant
@rivy
0