8000
Under-reporting of coverages (more generally, lack of monitoring tool interoperability) · Issue #333 · pyutils/line_profiler · GitHub
More Web Proxy on the site http://driver.im/
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
(EDIT: my finger slipped and submitted the issue before I'm finished typing. A moment please before I fix it up incrementally...)Done as of UTC 2025-04-14 09:16+00:00
Synopsis
We're vying over the control for sys.settrace() with coverage.py,1 which results in coverage data ceasing being collected from each Python process once the first LineProfiler instance has been .enable()-ed. There are multiple strategies we can employ to mitigate this, each with their advantages and drawbacks.
The current state
One thing that has bugged devs for the repo for a long time is that coverage.py is behaving unexpectedly, under-reporting on test coverage and neglecting code paths that has clearly been executed. This has the side effect of cluttering codecov output and PR diffs, since they are polluted with false alarms – to the point that coverage reports may cause more confusion and annoyance2 than they offer insights.
While that is unfortunately the current state of affairs, I still believe that coverage.py is invaluable as a QA tool and we can maybe try harder to fix it. So I took a deeper look and...
The reason
Every time we call LineProfiler.enable(), we PyEval_SetTrace() with a C-level Py_tracefunc and the profiler object (_line_profiler.pyx (L338))
Every time we call LineProfiler.disable(), we completely purge the tracing facilities by nulling the pointers to both the Py_tracefunc and the tracing object (unset_trace.c (L6)).
Hence whenever we run an in-process test which uses a LineProfiler, coverage.py only sees up to the point that the first LineProfiler is enabled, and the coverage-tracing function isn't restored even after the profiler has been disabled. The fundamental issue here is that tracing tools all have to use the same sys.settrace(), and Python doesn't natively provide for a way for tools to work cooperatively.
How to mitigate?
Naïve Python implementation
line_profiler/line_profiler.py::LineProfiler.enable() can sys.gettrace() to get the current tracer, and then .disable() can sys.settrace() and put it back.
PROS
It's simple and elegant. It also works on the Python level, so no need to go spelunking in _line_profiler.pyx.
CONS
It simply DOESN'T WORK in practice, since it only "works" for tracers set in Python-space and not in C-space. Specifically:
If a tracer is set on the C-level (PyEval_SetTrace(Py_tracefunc func, PyObject *obj)), sys.gettrace() would only retrieve obj and silently drop all info related to func.
And then when one proceeds to sys.settrace() to "restore the previous tracer", sys.settrace() supplies a default Py_tracefunc (Python/sysmodule.c::trace_trampoline() (L1101)) which essentially just calls the obj.3
Coverage-tracing is disabled as long as the profiler is active.
C implementation
line_profiler/_line_profiler.pyx::LineProfiler.enable() can retrieve references to both the Py_tracefunc and the tracer object, stash them somewhere, and restore them in .disable().
PROS
It's more robust, working for both pure-Python and C-level tracers.
CONS
There is no public C API for retrieving the Py_tracefunc: we'll have to hack into the thread state with PyThreadState_Get() and get the non-public member ->c_tracefunc (see Python/legacy_tracing.c::setup_tracing() (L585); meanwhile, sys.gettrace() retrieves the ->c_traceobj).
Coverage-tracing is still disabled as long as the profiler is active.
C-wrapper implementation
line_profiler/_line_profiler.pyx::LineProfiler.enable() can retrieve references to both the Py_tracefunc callback and the tracer object, and stash them somewhere.
line_profiler/_line_profiler.pyx::python_trace_callback() then retrieves said references from the LineProfiler object, and calls them on exit.
The old tracer object and callback are to be restored in LineProfiler.disable().
PROS
It allows for interoperability between LineProfiler and other monitoring toolings.
CONS
It's the most complex of the three solutions.
Such wrapping of other tracers may not be idiomatic usage and may cause unforeseen issues.
Footnotes
sys.monitoring (API; our implementation) is supposed to alleviate some of this by signaling that tracing facilities are in use, preventing tools from stepping over each others' toes. Notably, each compliant tool registers itself with sys.monitoring to get a soft hold over a tool ID, but since we're a profiler with ID .PROFILER_ID = 5 while coverage.py has .COVERAGE_ID = 1. However, there's only so much that it can do because since fundamentally tools with different IDs still have to use the same sys.settrace(), PyEval_SetTrace(), etc., and there can only be one tracing callback. ↩
I've been burnt by this in a line_profiler-based project I'm working on. The resultant stack traces were... baffling and borderline un-tractable to say the least. ↩
The text was updated successfully, but these errors were encountered:
TTsangSC
changed the title
Under-reporting of coverages
Under-reporting of coverages (more generally, lack of monitoring tool interoperability)
Apr 14, 2025
(EDIT: my finger slipped and submitted the issue before I'm finished typing. A moment please before I fix it up incrementally...)Done as of UTC 2025-04-14 09:16+00:00Synopsis
We're vying over the control for
sys.settrace()withcoverage.py,1 which results in coverage data ceasing being collected from each Python process once the firstLineProfilerinstance has been.enable()-ed. There are multiple strategies we can employ to mitigate this, each with their advantages and drawbacks.The current state
One thing that has bugged devs for the repo for a long time is that
coverage.pyis behaving unexpectedly, under-reporting on test coverage and neglecting code paths that has clearly been executed. This has the side effect of clutteringcodecovoutput and PR diffs, since they are polluted with false alarms – to the point that coverage reports may cause more confusion and annoyance2 than they offer insights.While that is unfortunately the current state of affairs, I still believe that
coverage.pyis invaluable as a QA tool and we can maybe try harder to fix it. So I took a deeper look and...The reason
LineProfiler.enable(), wePyEval_SetTrace()with a C-levelPy_tracefuncand the profiler object (_line_profiler.pyx(L338))LineProfiler.disable(), we completely purge the tracing facilities by nulling the pointers to both thePy_tracefuncand the tracing object (unset_trace.c(L6)).Hence whenever we run an in-process test which uses a
LineProfiler,coverage.pyonly sees up to the point that the firstLineProfileris enabled, and the coverage-tracing function isn't restored even after the profiler has been disabled. The fundamental issue here is that tracing tools all have to use the samesys.settrace(), and Python doesn't natively provide for a way for tools to work cooperatively.How to mitigate?
Naïve Python implementation
line_profiler/line_profiler.py::LineProfiler.enable()cansys.gettrace()to get the current tracer, and then.disable()cansys.settrace()and put it back.PROS
It's simple and elegant. It also works on the Python level, so no need to go spelunking in
_line_profiler.pyx.CONS
PyEval_SetTrace(Py_tracefunc func, PyObject *obj)),sys.gettrace()would only retrieveobjand silently drop all info related tofunc.sys.settrace()to "restore the previous tracer",sys.settrace()supplies a defaultPy_tracefunc(Python/sysmodule.c::trace_trampoline()(L1101)) which essentially just calls theobj.3C implementation
line_profiler/_line_profiler.pyx::LineProfiler.enable()can retrieve references to both thePy_tracefuncand the tracer object, stash them somewhere, and restore them in.disable().PROS
It's more robust, working for both pure-Python and C-level tracers.
CONS
Py_tracefunc: we'll have to hack into the thread state withPyThreadState_Get()and get the non-public member->c_tracefunc(seePython/legacy_tracing.c::setup_tracing()(L585); meanwhile,sys.gettrace()retrieves the->c_traceobj).C-wrapper implementation
line_profiler/_line_profiler.pyx::LineProfiler.enable()can retrieve references to both thePy_tracefunccallback and the tracer object, and stash them somewhere.line_profiler/_line_profiler.pyx::python_trace_callback()then retrieves said references from theLineProfilerobject, and calls them on exit.LineProfiler.disable().PROS
It allows for interoperability between
LineProfilerand other monitoring toolings.CONS
Footnotes
sys.monitoring(API; our implementation) is supposed to alleviate some of this by signaling that tracing facilities are in use, preventing tools from stepping over each others' toes. Notably, each compliant tool registers itself withsys.monitoringto get a soft hold over a tool ID, but since we're a profiler with ID.PROFILER_ID = 5whilecoverage.pyhas.COVERAGE_ID = 1. However, there's only so much that it can do because since fundamentally tools with different IDs still have to use the samesys.settrace(),PyEval_SetTrace(), etc., and there can only be one tracing callback. ↩Exhibit A; exhibit B; exhibit C ↩
I've been burnt by this in a
line_profiler-based project I'm working on. The resultant stack traces were... baffling and borderline un-tractable to say the least. ↩The text was updated successfully, but these errors were encountered: