FIX: Update hash tables of affected profiler instances when rewriting code objects #351
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8000
line_profiler/_line_profiler.pyx::LineProfiler.get_stats()
Updated/simplified implementation to aggregate profiling data
between code objects with duplicate hashes
line_profiler/_line_profiler.pyx::LineProfiler
_all_instances_by_bcodes
New private class attribute for keeping track of:
- Instances profiling the same bytecode
- Count for how the the bytecode should be padded
dupes_map
Now containing the up-to-date (instead of pre-padding) code objs
add_function()
Refactored implementation to:
- Pad bytecodes according to `._all_instances_by_bcodes` instead
of the instance's `.dupes_map`
- Use `.dupes_map` to identify profiler instances profiling the
same functions and update their hash tables
- Curate `.dupes_map` on said instances to stay up-to-date
line_profiler/_line_profiler.pyx::LineProfiler
add_function()
- Now identifying update targets based on function-object
identity instead of code-object identity (because different
function objects can share the same code object)
- Simplified implementation
_all_instances_by_bcodes
Superseded by other private class attributes:
- `._all_paddings`:
mapping from bytecodes to padding length
- `._all_instances_by_funcs`:
mapping from function identity to `WeakSet`s of instances
tests/test_line_profiler.py
test_multiple_profilers_identical_bytecode()
New test for applying multiple profilers to identical functions
in arbitrary order (one subtest currently failing)
line_profiler/_line_profiler.pyx::LineProfiler.add_function()
Now stripping padded code objects and re-padding when appropriate,
so that no two profiled code objects can end up with the same
bytecode
tests/test_line_profiler.py
test_multiple_profilers_identical_bytecode()
- Updated comments and docstring
- Added check against duplicate bytecodes
tests/test_line_profiler.py
get_prof_stats()
New helper function for formatting and retrieving
`LineProfiler.print_stats()` output
test_*_decorator()
test_profile_generated_code()
test_multiple_profilers_identical_bytecode()
Refactored to use `get_prof_stats()`
test_aggregate_profiling_data_between_code_versions()
New test checking that profiling data gathered before and after
a function's code object is rewritten are correctly aggregated
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #351 +/- ##
=======================================
Coverage 65.33% 65.33%
=======================================
Files 13 13
Lines 1073 1073
Branches 234 234
=======================================
Hits 701 701
Misses 310 310
Partials 62 62 Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #350. Includes #349, which by extension closes #348. New changes are from 7338799 onwards.
Code changes
line_profiler/_line_profiler.pyx::LineProfiler.add_function().dupes_mapitems for padded bytecodes._all_paddings,._all_instances_by_funcsNew private class attributes for use by
.add_function()Test changes
tests/test_line_profiler.pytest_{{class,static,bound,partial}method,partial,[cached_]property}_decorator(),test_profile_generated_code()Minor refactoring with a utility function for more readable output
test_multiple_profilers_identical_bytecode()New test checking that the order in which profilers are used to decorate functions shouldn't affect the output (up to overhead)
test_aggregate_profiling_data_between_code_versions()New test checking that existing profiling data should persist and be aggregated with new data after another profiler replaced the code object of a profiled function
Performance impact
Test suite run 5 times with the command
pytest 'not (test_duplicate or test_aggregate or identical_bytecode)', i.e. excluding the tests added by this PR (and #349); impact on performance should be minimal.mainmulti-desync-fix(this PR)