8000 Unit tests fail depending on how many are being run · Issue #273 · NVIDIA/NeMo-text-processing · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Unit tests fail depending on how many are being run #273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
i-am-bbking opened this issue Apr 29, 2025 · 1 comment
Open

Unit tests fail depending on how many are being run #273

i-am-bbking opened this issue Apr 29, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@i-am-bbking
Copy link
i-am-bbking commented Apr 29, 2025 •
edited
Loading

Describe the bug

Unit tests fail depending on how many are being run.

Steps/Code to reproduce bug

When I run this in main branch, 5 tests fail:

$ pytest --verbose  --cpu tests/nemo_text_processing/fr/
.
.
.

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestDate.test_norm_00_02_03_2003 ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

a = (<tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cef7f0>,), kw = {}

    @wraps(func)
    def standalone_func(*a, **kw):
>       return func(*(a + p.args), **p.kwargs, **kw)

../../miniconda3/envs/nemo_tn/lib/python3.10/site-packages/parameterized/parameterized.py:620: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cef7f0>, test_input = '02.03.2003', expected = 'deux mars deux mille trois'

    @parameterized.expand(parse_test_case_file('fr/data_text_normalization/test_cases_date.txt'))
    @pytest.mark.run_only_on('CPU')
    @pytest.mark.unit
    def test_norm(self, test_input, expected):
        pred = self.normalizer.normalize(test_input, verbose=False)
>       assert pred == expected
E       AssertionError: assert 'zéro deux . zéro trois . deux mille trois' == 'deux mars deux mille trois'
E         
E         - deux mars deux mille trois
E         + zéro deux . zéro trois . deux mille trois

tests/nemo_text_processing/fr/test_date.py:41: AssertionError

 tests/nemo_text_processing/fr/test_date.py::TestDate.test_norm_00_02_03_2003 ⨯                                                                                                                                                      30% ███       

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestDate.test_norm_01_02_03_2003 ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

a = (<tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cfd540>,), kw = {}

    @wraps(func)
    def standalone_func(*a, **kw):
>       return func(*(a + p.args), **p.kwargs, **kw)

../../miniconda3/envs/nemo_tn/lib/python3.10/site-packages/parameterized/parameterized.py:620: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cfd540>, test_input = '02/03/2003', expected = 'deux mars deux mille trois'

    @parameterized.expand(parse_test_case_file('fr/data_text_normalization/test_cases_date.txt'))
    @pytest.mark.run_only_on('CPU')
    @pytest.mark.unit
    def test_norm(self, test_input, expected):
        pred = self.normalizer.normalize(test_input, verbose=False)
>       assert pred == expected
E       AssertionError: assert 'zéro deux zéroièmes trois deux mille troisièmes' == 'deux mars deux mille trois'
E         
E         - deux mars deux mille trois
E         + zéro deux zéroièmes trois deux mille troisièmes

tests/nemo_text_processing/fr/test_date.py:41: AssertionError

 tests/nemo_text_processing/fr/test_date.py::TestDate.test_norm_01_02_03_2003 ⨯                                                                                                                                                      30% ███       

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestDate.test_norm_02_02_03_2003 ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

a = (<tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cfe560>,), kw = {}

    @wraps(func)
    def standalone_func(*a, **kw):
>       return func(*(a + p.args), **p.kwargs, **kw)

../../miniconda3/envs/nemo_tn/lib/python3.10/site-packages/parameterized/parameterized.py:620: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cfe560>, test_input = '02-03-2003', expected = 'deux mars deux mille trois'

    @parameterized.expand(parse_test_case_file('fr/data_text_normalization/test_cases_date.txt'))
    @pytest.mark.run_only_on('CPU')
    @pytest.mark.unit
    def test_norm(self, test_input, expected):
        pred = self.normalizer.normalize(test_input, verbose=False)
>       assert pred == expected
E       AssertionError: assert 'zéro deux moins zéro trois moins deux mille trois' == 'deux mars deux mille trois'
E         
E         - deux mars deux mille trois
E         + zéro deux moins zéro trois moins deux mille trois

tests/nemo_text_processing/fr/test_date.py:41: AssertionError

 tests/nemo_text_processing/fr/test_date.py::TestDate.test_norm_02_02_03_2003 ⨯                                                                                                                                                      30% ███▏      

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestDate.test_norm_03_le_02_03_2003 ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

a = (<tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cfeb60>,), kw = {}

    @wraps(func)
    def standalone_func(*a, **kw):
>       return func(*(a + p.args), **p.kwargs, **kw)

../../miniconda3/envs/nemo_tn/lib/python3.10/site-packages/parameterized/parameterized.py:620: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cfeb60>, test_input = 'le 02.03.2003', expected = 'le deux mars deux mille trois'

    @parameterized.expand(parse_test_case_file('fr/data_text_normalization/test_cases_date.txt'))
    @pytest.mark.run_only_on('CPU')
    @pytest.mark.unit
    def test_norm(self, test_input, expected):
        pred = self.normalizer.normalize(test_input, verbose=False)
>       assert pred == expected
E       AssertionError: assert 'le zéro deux . zéro trois . deux mille trois' == 'le deux mars deux mille trois'
E         
E         - le deux mars deux mille trois
E         + le zéro deux . zéro trois . deux mille trois

tests/nemo_text_processing/fr/test_date.py:41: AssertionError

 tests/nemo_text_processing/fr/test_date.py::TestDate.test_norm_03_le_02_03_2003 ⨯                                                                                                                                                   31% ███▏      

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestDate.test_norm_04_17_06 ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

a = (<tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cfed70>,), kw = {}

    @wraps(func)
    def standalone_func(*a, **kw):
>       return func(*(a + p.args), **p.kwargs, **kw)

../../miniconda3/envs/nemo_tn/lib/python3.10/site-packages/parameterized/parameterized.py:620: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.nemo_text_processing.fr.test_date.TestDate object at 0x107cfed70>, test_input = '17.06', expected = 'dix-sept juin'

    @parameterized.expand(parse_test_case_file('fr/data_text_normalization/test_cases_date.txt'))
    @pytest.mark.run_only_on('CPU')
    @pytest.mark.unit
    def test_norm(self, test_input, expected):
        pred = self.normalizer.normalize(test_input, verbose=False)
>       assert pred == expected
E       AssertionError: assert 'dix-sept . zéro six' == 'dix-sept juin'
E         
E         - dix-sept juin
E         + dix-sept . zéro six

tests/nemo_text_processing/fr/test_date.py:41: AssertionError

Results (147.45s (0:02:27)):
     419 passed
       5 failed
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_00_02_03_2003
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_01_02_03_2003
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_02_02_03_2003
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_03_le_02_03_2003
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_04_17_06

However, when I run a smaller set that includes the failed tests, 0 tests fail:

pytest --verbose  --cpu tests/nemo_text_processing/fr/test_date.py
Test session starts (platform: darwin, Python 3.10.13, pytest 8.3.5, pytest-sugar 1.0.0)
cachedir: .pytest_cache
configfile: setup.cfg
plugins: anyio-4.9.0, sugar-1.0.0, progress-1.3.0, typeguard-4.4.2, xdist-3.6.1
collected 19 items                                                                                                                                                                                                                                

Results (17.66s):
      19 passed

Expected behavior

I expect unit tests to succeed in both runs.

Environment overview (please complete the following information)

  • Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
  • Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
  • If method of install is [Docker], provide docker pull & docker run commands used

Environment details

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:

  • OS version: macOS 15.4.1 (24E263)
  • PyTorch version: 2.6.0
  • Python version: Python 3.10.13

Additional context

This happens with other subsets of runs as well.
@i-am-bbking i-am-bbking added the bug Something isn't working label Apr 29, 2025
@i-am-bbking
Copy link
Author

I've narrowed down the problem a bit.

This runs without any failures:

pytest --verbose  --cpu --tn_cache_dir ~/tn_cache_dir tests/nemo_text_processing/fr/
...
Results (2.73s):
     424 passed

However, running this has failures.

pytest --verbose  --cpu tests/nemo_text_processing/fr/
...
Results (118.30s (0:01:58)):
     419 passed
       5 failed
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_00_02_03_2003
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_01_02_03_2003
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_02_02_03_2003
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_03_le_02_03_2003
         - tests/nemo_text_processing/fr/test_date.py:36 TestDate.test_norm_04_17_06

Ideally, the unit tests would be less brittle and succeed in both conditions. If that is not possible, then there should be clear documentation how the unit tests should be run, with the best version enabled by default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@i-am-bbking
0