Support both train / eval modes for ModuleInfo #78735
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
🔗 Helpful links
✅ No Failures (46 Pending)As of commit 6d46096 (more details on the Dr. CI page): Expand to see more💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Please report bugs/suggestions to the (internal) Dr. CI Users group. |
This PR enhances the tests in `test/test_modules.py` to be run across train / eval modes. It allows sample inputs to be generated based on the mode under test (useful if e.g. if a certain sample should only be used during eval mode). [ghstack-poisoned]
This PR enhances the tests in `test/test_modules.py` to be run across train / eval modes. It makes the following changes:
* Adds a required `training` arg to `ModuleInfo.module_inputs_func` - this allows those functions to generate sample module inputs based on the training mode.
* Updates the `modules` decorator to additionally pass a `training` arg to test functions.
* The `training` arg values to pass can be customized via `TrainEvalMode` e.g. `modules(..., train_eval_mode=TrainEvalMode.train_only)`. Supported modes are `train_only`, `eval_only`, and `train_and_eval`.
* Tests that use the decorator in `test/test_modules.py` have been updated. They now ingest the `training` arg, pass it to the `module_inputs_func`, and set the training mode for instantiated module instances with `m.train(training)`. There's almost certainly a way to do this with less boilerplate.
* Adds a new `train_and_eval_differ` option to `ModuleInfo` for indicating whether a specific module cares about the training mode.
* If this is `False`, tests are generated just for training by default. This avoids pointlessly running tests twice when the training mode setting doesn't matter.
* Adds a new test in `test/test_modules.py` to verify that `train_and_eval_differ` is set correctly. It does this by instantiating each module, deleting the `training` attribute, and then running the forward pass. If the forward pass references `training`, it will throw an `AttributeError`, indicating that `train_and_eval_differ=True` should be set.
[ghstack-poisoned]
albanD
approved these changes
Jun 8, 2022
jbschlosser
commented
Jun 8, 2022
There was a problem hiding this comment.
@albanD Before I can merge this, it looks like I need a way to skip a test for eval mode only.
I have a working fix that can skip based on anything passed to the test - going to add a commit earlier in the PR stack that adds this
This PR enhances the tests in `test/test_modules.py` to be run across train / eval modes. It makes the following changes:
* Adds a required `training` arg to `ModuleInfo.module_inputs_func` - this allows those functions to generate sample module inputs based on the training mode.
* Updates the `modules` decorator to additionally pass a `training` arg to test functions.
* The `training` arg values to pass can be customized via `TrainEvalMode` e.g. `modules(..., train_eval_mode=TrainEvalMode.train_only)`. Supported modes are `train_only`, `eval_only`, and `train_and_eval`.
* Tests that use the decorator in `test/test_modules.py` have been updated. They now ingest the `training` arg, pass it to the `module_inputs_func`, and set the training mode for instantiated module instances with `m.train(training)`. There's almost certainly a way to do this with less boilerplate.
* Adds a new `train_and_eval_differ` option to `ModuleInfo` for indicating whether a specific module cares about the training mode.
* If this is `False`, tests are generated just for a single mode. This avoids pointlessly running tests twice when the training mode setting doesn't matter.
* Adds a new test in `test/test_modules.py` to verify that `train_and_eval_differ` is set correctly. It does this by instantiating each module, deleting the `training` attribute, and then running the forward pass. If the forward pass references `training`, it will throw an `AttributeError`, indicating that `train_and_eval_differ=True` should be set.
[ghstack-poisoned]
|
@pytorchbot merge -g |
albanD
approved these changes
Jun 8, 2022
|
@pytorchbot successfully started a merge job. Check the current status here |
|
Hey @jbschlosser. |
|
@pytorchbot revert -m "Broke eval tests on Win, 10.2 and ROCM, see https://hud.pytorch.org/pytorch/pytorch/commit/12658fcd5bdf4d2437754633b3fa39ab15d213b9 " -c missingsignal |
|
❌ 🤖 pytorchbot command failed: Try |
|
@pytorchbot revert -m "Broke eval tests on Win, 10.2 and ROCM, see https://hud.pytorch.org/pytorch/pytorch/commit/12658fcd5bdf4d2437754633b3fa39ab15d213b9 " -c nosignal |
|
@pytorchbot successfully started a revert job. Check the current status here |
pytorchmergebot
added a commit
that referenced
this pull request
Jun 9, 2022
This reverts commit 12658fc. Reverted #78735 on behalf of https://github.com/malfet due to Broke eval tests on Win, 10.2 and ROCM, see https://hud.pytorch.org/pytorch/pytorch/commit/12658fcd5bdf4d2437754633b3fa39ab15d213b9
This PR enhances the tests in `test/test_modules.py` to be run across train / eval modes. It makes the following changes:
* Adds a required `training` arg to `ModuleInfo.module_inputs_func` - this allows those functions to generate sample module inputs based on the training mode.
* Updates the `modules` decorator to additionally pass a `training` arg to test functions.
* The `training` arg values to pass can be customized via `TrainEvalMode` e.g. `modules(..., train_eval_mode=TrainEvalMode.train_only)`. Supported modes are `train_only`, `eval_only`, and `train_and_eval`.
* Tests that use the decorator in `test/test_modules.py` have been updated. They now ingest the `training` arg, pass it to the `module_inputs_func`, and set the training mode for instantiated module instances with `m.train(training)`. There's almost certainly a way to do this with less boilerplate.
* Adds a new `train_and_eval_differ` option to `ModuleInfo` for indicating whether a specific module cares about the training mode.
* If this is `False`, tests are generated just for a single mode. This avoids pointlessly running tests twice when the training mode setting doesn't matter.
* Adds a new test in `test/test_modules.py` to verify that `train_and_eval_differ` is set correctly. It does this by instantiating each module, deleting the `training` attribute, and then running the forward pass. If the forward pass references `training`, it will throw an `AttributeError`, indicating that `train_and_eval_differ=True` should be set.
[ghstack-poisoned]
|
@pytorchbot merge -a |
|
@pytorchbot successfully started a merge job. Check the current status here |
|
Merge failed due to Refusing to merge as mandatory check(s) linux-docs / build-docs (cpp) are pending/not yet run for rule superuser |
|
@pytorchbot merge -a |
|
@pytorchbot successfully started a merge job. Check the current status here |
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 10, 2022
Summary: Pull Request resolved: #78735 Approved by: https://github.com/albanD Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/12658fcd5bdf4d2437754633b3fa39ab15d213b9 Reviewed By: osalpekar Differential Revision: D37025769 Pulled By: jbschlosser fbshipit-source-id: 4c9c842f4bc6dbed3a37fd464dfbad2fe48bad17
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 10, 2022
Summary: This reverts commit 12658fc. Reverted #78735 on behalf of https://github.com/malfet due to Broke eval tests on Win, 10.2 and ROCM, see https://hud.pytorch.org/pytorch/pytorch/commit/12658fcd5bdf4d2437754633b3fa39ab15d213b9 Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/854c833f8101b62ccda9198dd0a66555e7caaa69 Reviewed By: osalpekar Differential Revision: D37030370 Pulled By: osalpekar fbshipit-source-id: 77810affd0ac9acefd8ea96a3d39781a0d5b9eaa
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 10, 2022
Summary: Pull Request resolved: #78735 Approved by: https://github.com/albanD Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/70d6446a3def513b8082d243d7996ef86c2787a6 Reviewed By: osalpekar Differential Revision: D37059361 Pulled By: osalpekar fbshipit-source-id: a179f93c722112a70bdcecc67f6c5d1c9234eb50
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
This PR enhances the tests in
test/test_modules.pyto be run across train / eval modes. It makes the following changes:trainingarg toModuleInfo.module_inputs_func- this allows those functions to generate sample module inputs based on the training mode.@modulesdecorator to additionally pass atrainingarg to test functions.trainingarg values to pass can be customized viaTrainEvalModee.g.@modules(..., train_eval_mode=TrainEvalMode.train_only). Supported modes aretrain_only,eval_only, andtrain_and_eval.test/test_modules.pyhave been updated. They now ingest thetrainingarg, pass it to themodule_inputs_func, and set the training mode for instantiated module instances withm.train(training). There's almost certainly a way to do this with less boilerplate.train_and_eval_differoption toModuleInfofor indicating whether a specific module cares about the training mode.False, tests are generated just for a single mode. This avoids pointlessly running tests twice when the training mode setting doesn't matter.test/test_modules.pyto verify thattrain_and_eval_differis set correctly. It does this by instantiating each module, deleting thetrainingattribute, and then running the forward pass. If the forward pass referencestraining, it will throw anAttributeError, indicating thattrain_and_eval_differ=Trueshould be set.