8000 Support more granular test decoration · Issue #79161 · pytorch/pytorch · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Support more granular test decoration #79161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jbschlosser opened this issue Jun 8, 2022 · 4 comments
Closed

Support more granular test decoration #79161

jbschlosser opened this issue Jun 8, 2022 · 4 comments
Labels
module: testing Issues related to the torch.testing module (not tests) triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@jbschlosser
Copy link
Contributor

Currently, the DecorateInfo class only supports conditionally applying decorators to test functions based on test class name, base test name, device, and dtype. Example usage:

pytorch/torch/testing/_internal/common_methods_invocations.py

Lines 9800 to 9805 in 7cb4a76

skips=(
# Ref: https://github.com/pytorch/pytorch/issues/77946
DecorateInfo(unittest.skip("Unable to reproduce failure locally"), "TestForeach",
"test_binary_op_scalarlist_fastpath",
device_type='cuda', dtypes=(torch.float16,)),
)

It would be useful to support more granular test decoration, ideally based on any parameter that is passed to the test function rather than just device / dtype.

Case in point: #78735 is adding support for running tests across train / eval modes for ModuleInfo. Some tests are expected to fail for eval mode only, and there's no granular way to indicate this.
@jbschlosser jbschlosser added the module: testing Issues related to the torch.testing module (not tests) label Jun 8, 2022
@zou3519 zou3519 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jun 9, 2022
@zou3519
Copy link
Contributor
zou3519 commented Jun 9, 2022

This seems difficult to add, what would the API look like? Would the user provide a lambda that, on accepting the parameters, return a decorator?

@kshitij12345
Copy link
Collaborator
kshitij12345 commented Jun 10, 2022 •
edited
Loading

I think this will also help with the usage of @parametrize once they become frequent.

pytorch/test/test_ops.py

Lines 515 to 518 in a299a2f

@skipCUDAIfRocm
@ops(python_ref_db)
@parametrize('executor', ['aten', 'nvfuser'])
def test_python_ref_executor(self, device, dtype, op, executor):

For the test above, if an operator fails only for one of the executor, we have no choice but to add skip. (With expectedFailure, the other executor passes and you get unexpected success)

@jbschlosser
Copy link
Contributor Author

@zou3519 I've gotten a PoC working locally that adds the support by enhancing the active_if arg that can be used with DecorateInfo. Usage is similar to what you've described - a lambda that accepts a dict with the params being passed to the test -> returning a bool for whether the DecorateInfo is active.

For the example from @kshitij12345, the relevant OpInfo entry would contain something like this:

DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),

In the current code, the test decoration / skipping mechanisms are implemented somewhat redundantly in OpInfo and ModuleInfo, although they both use DecorateInfo. To support decoration based on any test param, I made this mechanism global instead so it's aware of the final state of the test function after composing decorators, etc. In my PoC, it's handled during test instantiation by instantiate_parametrized_tests() / instantiate_device_type_tests().

A nice side effect of making the mechanism global is that it opens up the possibility for usage outside of @ops / @modules as well. It allows for things like this:

# Example: Expect failure if x == 3 and we're in eval mode.
@applyIf(unittest.expectedFailure,
         lambda params: params['x'] == 3 and not params['training'])
@parametrize("x", range(5))
@parametrize("training", [False, True])
def test_foo(self, x, training):
    ...

Open to bikeshedding on how the API could look :)

@lezcano
Copy link
Collaborator
lezcano commented Jun 16, 2022

Note that this would be very useful for regular OpInfos.

There are some operations that have valid outputs, but that are not differentiable on those outputs. The most recent example I've bumped into is logdet which returns NaN if the input is real and has negative determinant. We could potentially skip gradchecks for these inputs, while still test them in all the other tests without having ot write and maintain a new OpInfo for them.

This was referenced Jun 16, 2022
Closed
Closed
jbschlosser added a commit that referenced this issue Jun 21, 2022
@jbschlosser
Update on "Support per-parameter test decoration"
b0179eb 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jun 24, 2022
@jbschlosser
Update on "Support per-parameter test decoration"
0597cf3 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jun 24, 2022
@jbschlosser
Update on "Support per-parameter test decoration"
0c883e3 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decor
10000
ator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jun 28, 2022
@jbschlosser
Update base for Update on "Support per-parameter test decoration"
b760726 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jun 28, 2022
@jbschlosser
Update on "Support per-parameter test decoration"
67286a7 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jun 28, 2022
@jbschlosser
Update base for Update on "Support per-parameter test decoration"
19ab45d 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jun 28, 2022
@jbschlosser
Update on "Support per-parameter test decoration"
f2fd307 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jun 29, 2022
@jbschlosser
Update base for Update on "Support per-parameter test decoration"
703c7a6 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jun 29, 2022
@jbschlosser
Update on "Support per-parameter test decoration"
d9c7ea1 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jul 8, 2022
@jbschlosser
Update base for Update on "Support per-parameter test decoration"
d679315 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jul 8, 2022
@jbschlosser
Update on "Support per-parameter test decoration"
d6add3c 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
@kshitij12345 kshitij12345 mentioned this issue Nov 17, 2022
Closed
@jbschlosser jbschlosser mentioned this issue Jan 3, 2023
Closed
jbschlosser added a commit that referenced this issue Jan 3, 2023
@jbschlosser
Support per-parameter test decoration
2eef52b 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jan 3, 2023
@jbschlosser
Support per-parameter test decoration
1e2fe94 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

ghstack-source-id: a1f28ba
Pull Request resolved: #91658
jbschlosser added a commit that referenced this issue Jan 4, 2023
@jbschlosser
Update on "Support per-parameter test decoration"
92965fe 
Continuation of #79979.

Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize`, `modules`, and `ops`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jan 4, 2023
@jbschlosser
Update on "Support per-parameter test decoration"
0d046af 
Continuation of #79979.

Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize`, `modules`, and `ops`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

[ghstack-poisoned]
jbschlosser added a commit that referenced this issue Jan 4, 2023
@jbschlosser
Support per-parameter test decoration
39f40ef 
Fixes #79161

This PR does the following:
* Expands the `parametrize_fn()` signature from returning a 3-tuple of `(test, test_name, param_kwargs)` to returning a 4-tuple of `(test, test_name, param_kwargs, decorator_fn)`. Expected signature for the addition is `decorator_fn(param_kwargs) -> List[decorator]` i.e. given the full set of test params, return a list of decorators to apply.
    * `modules`, `ops`, and `parametrize` now fit the new signature, returning `decorator_fn`s instead of applying decorators themselves.
    * `instantiate_parametrized_tests()` and `instantiate_device_type_tests()` now call the returned `decorator_fn`, passing in the full set of `param_kwargs` (after composition + `device` / `dtype` additions) and applying the returned decorators.
    * Composing multiple `parametrize_fn`s also composes the corresponding `decorator_fn`s; the composed `decorator_fn` simply concatenates the decorator lists returned by the constituents.
* Expands `DecorateInfo.is_active` to support callables:
```python
DecorateInfo(
    unittest.expectedFailure, "TestOps", "test_python_ref_executor",
    device_type='cuda', active_if=lambda params: params['executor'] == 'nvfuser'
),
```
* Adds several tests to `test/test_testing.py` ensuring proper decoration using `parametrize` and `modules`.
* (minor) Refactors `ModuleInfo` slightly to better match `OpInfo`; replace `should_skip()` with `get_decorators()` and merge `skips` and `decorators`.
* (minor) Fixes a couple `ModuleInfo` naming oddities uncovered during testing.

ghstack-source-id: 0f63326
Pull Request resolved: #91658
@jbschlosser jbschlosser mentioned this issue Oct 25, 2023
Closed
pytorchmergebot pushed a commit that referenced this issue Oct 26, 2023
@jbschlosser @pytorchmergebot
New @decorateIf decorator for param-specific conditional decoration (#…
42e4c64 
…112033)

Adds a new decorator `@decorateIf(decorator, predicate_fn)`. Examples:
```python
from torch.testing._internal.common_utils import decorateIf
...

@decorateIf(unittest.skip, lambda params: params["x"] == 2)
@parametrize("x", range(5))
def test_foo(self, x):
    ...

@parametrize("x,y", [(1, 'foo'), (2, 'bar'), (3, 'baz')])
@decorateIf(
    unittest.expectedFailure,
    lambda params: params["x"] == 3 and params["y"] == "baz"
)
def test_bar(self, x, y):
    ...

@decorateIf(
    unittest.expectedFailure,
    lambda params: params["op"].name == "add" and params["dtype"] == torch.float16
)
@ops(op_db)
def test_op_foo(self, device, dtype, op):
    ...

@decorateIf(
    unittest.skip,
    lambda params: params["module_info"].module_cls is torch.nn.Linear and \
        params["device"] == "cpu"
)
@modules(module_db)
def test_module_foo(self, device, dtype, module_info):
    ...
```

Follow-up for per-param decoration based on #79161 (comment)
Pull Request resolved: #112033
Approved by: https://github.com/clee2000, https://github.com/pmeier
xuhancn pushed a commit to xuhancn/pytorch that referenced this issue Nov 7, 2023
@jbschlosser @xuhancn
New @decorateIf decorator for param-specific conditional decoration (p…
9c9db54 
…ytorch#112033)

Adds a new decorator `@decorateIf(decorator, predicate_fn)`. Examples:
```python
from torch.testing._internal.common_utils import decorateIf
...

@decorateIf(unittest.skip, lambda params: params["x"] == 2)
@parametrize("x", range(5))
def test_foo(self, x):
    ...

@parametrize("x,y", [(1, 'foo'), (2, 'bar'), (3, 'baz')])
@decorateIf(
    unittest.expectedFailure,
    lambda params: params["x"] == 3 and params["y"] == "baz"
)
def test_bar(self, x, y):
    ...

@decorateIf(
    unittest.expectedFailure,
    lambda params: params["op"].name == "add" and params["dtype"] == torch.float16
)
@ops(op_db)
def test_op_foo(self, device, dtype, op):
    ...

@decorateIf(
    unittest.skip,
    lambda params: params["module_info"].module_cls is torch.nn.Linear and \
        params["device"] == "cpu"
)
@modules(module_db)
def test_module_foo(self, device, dtype, module_info):
    ...
```

Follow-up for per-param decoration based on pytorch#79161 (comment)
Pull Request resolved: pytorch#112033
Approved by: https://github.com/clee2000, https://github.com/pmeier
Skylion007 pushed a commit to Skylion007/pytorch that referenced this issue Nov 14, 2023
@jbschlosser @Skylion007
New @decorateIf decorator for param-specific conditional decoration (p…
da54a97 
…ytorch#112033)

Adds a new decorator `@decorateIf(decorator, predicate_fn)`. Examples:
```python
from torch.testing._internal.common_utils import decorateIf
...

@decorateIf(unittest.skip, lambda params: params["x"] == 2)
@parametrize("x", range(5))
def test_foo(self, x):
    ...

@parametrize("x,y", [(1, 'foo'), (2, 'bar'), (3, 'baz')])
@decorateIf(
    unittest.expectedFailure,
    lambda params: params["x"] == 3 and params["y"] == "baz"
)
def test_bar(self, x, y):
    ...

@decorateIf(
    unittest.expectedFailure,
    lambda params: params["op"].name == "add" and params["dtype"] == torch.float16
)
@ops(op_db)
def test_op_foo(self, device, dtype, op):
    ...

@decorateIf(
    unittest.skip,
    lambda params: params["module_info"].module_cls is torch.nn.Linear and \
        params["device"] == "cpu"
)
@modules(module_db)
def test_module_foo(self, device, dtype, module_info):
    ...
```

Follow-up for per-param decoration based on pytorch#79161 (comment)
Pull Request resolved: pytorch#112033
Approved by: https://github.com/clee2000, https://github.com/pmeier
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
module: testing Issues related to the torch.testing module (not tests) triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lezcano @zou3519 @kshitij12345 @jbschlosser
0