feat: add tracing spans to ScanExec and TakeExec #3766
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3766 +/- ##
==========================================
- Coverage 78.60% 78.55% -0.05%
==========================================
Files 272 273 +1
Lines 101951 101933 -18
Branches 101951 101933 -18
==========================================
- Hits 80138 80075 -63
- Misses 18656 18690 +34
- Partials 3157 3168 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
LuQQiu
approved these changes
May 5, 2025
HaochengLIU
pushed a commit
to HaochengLIU/lance
that referenced
this pull request
May 7, 2025
* Fixes spans to record `object_store::path::Path` with `.as_ref()`, so
it displays as `"/my/path"` instead of `Path { inner: "/my/path/" }`.
* Add `size` fields to some object store spans
* Pass span does into the background IO loop of the scheduler. This way
we can see all IO calls associated with a query in its spans.
* Changes `read_deletion_file` span to only start if there is a deletion
file. This reduces span spam in the traces where there are no deletion
files.
* Change `apply_deletions` span to only start if there are deletions.
Similar to above, reduces span clutter in traces.
* Add spans to `MergeInsertJob::execute_uncommitted_impl` and
`CommitBuilder::execute` so we can clearly see the write and commit step
in each attempt of `MergeInsert` job retries.
* Add spans for `LanceScanExec::execute` and `TakeExec::execute`. Each
of these will also record the IO metrics as part of their span
attributes.
* perf: In V2 scans, use out-of-order buffering of read futures if the
user didn't request ordered output for the scan.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
object_store::path::Pathwith.as_ref(), so it displays as"/my/path"instead ofPath { inner: "/my/path/" }.sizefields to some object store spansread_deletion_filespan to only start if there is a deletion file. This reduces span spam in the traces where there are no deletion files.apply_deletionsspan to only start if there are deletions. Similar to above, reduces span clutter in traces.MergeInsertJob::execute_uncommitted_implandCommitBuilder::executeso we can clearly see the write and commit step in each attempt ofMergeInsertjob retries.LanceScanExec::executeandTakeExec::execute. Each of these will also record the IO metrics as part of their span attributes.