Logging (1/2): new delay detector #9540
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Sep 20, 2023
wacban
approved these changes
Sep 20, 2023
| use tracing_subscriber::Layer; | ||
|
|
||
| #[derive(Default)] | ||
| pub(crate) struct DelayDetectorLayer {} |
There was a problem hiding this comment.
How about TracingSpanDurationLogger or Subscriber?
There was a problem hiding this comment.
Thank you for the suggestion. SpanDurationLogger should be clear enough.
nagisa
reviewed
Sep 20, 2023
| let mut extensions = span.extensions_mut(); | ||
| if let Some(timings) = extensions.get_mut::<Timings>() { | ||
| let now = Instant::now(); | ||
| timings.idle += (now - timings.last).as_nanos() as u64; |
There was a problem hiding this comment.
Use Instant::duration_since, or preferably checked_duration_since. Although Instant should never go backwards, that unfortunately does not always hold experimentally, and we would rather not have neard crash because of a debugging feature.
There was a problem hiding this comment.
duration_since() doesn't panic given a timestamp later than self, but saturates: https://doc.rust-lang.org/std/time/struct.Instant.html#method.duration_since
| let span = ctx.span(&id).expect("Span not found, this is a bug"); | ||
| let extensions = span.extensions(); | ||
| if let Some(Timings { busy, mut idle, last }) = extensions.get::<Timings>() { | ||
| idle += (Instant::now() - *last).as_nanos() as u64; |
There was a problem hiding this comment.
I would perhaps add a method to Timings along the lines of
impl Timings {
fn observe_idle(&mut self) {
let previous = std::mem::replace(self.last, Instant::now());
self.idle += self.last.checked_duration_since(previous).map_or(0, |d| d.as_nanos());
}
}Thus reusing the logic between this and on_enter.
The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files.
The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files.
The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files.
nikurt
added a commit
that referenced
this pull request
Sep 22, 2023
* Logging: New delay detector * Logging: New delay detector * Logging: New delay detector * Logging: New delay detector * Rename to SpanDurationLogger * Use Durations instead of nanoseconds * refactor: Tracing (#9544) The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files. * Logging: New delay detector * Rename to SpanDurationLogger * Use Durations instead of nanoseconds * Use Durations instead of nanoseconds * refactor: Tracing (#9544) The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files. * Logging: New delay detector * refactor: Tracing (#9544) The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files. * Use Durations instead of nanoseconds
github-merge-queue bot
pushed a commit
that referenced
this pull request
Sep 25, 2023
Continues #9540 Deletes the `DelayDetector` class and replaces the uses of it with spans. --------- Co-authored-by: Anton Puhach <anton@near.org> Co-authored-by: Simonas Kazlauskas <git@kazlauskas.me> Co-authored-by: Jakob Meier <mail@jakobmeier.ch>
nikurt
added a commit
that referenced
this pull request
Sep 26, 2023
* Logging: New delay detector * Logging: New delay detector * Logging: New delay detector * Logging: New delay detector * Rename to SpanDurationLogger * Use Durations instead of nanoseconds * refactor: Tracing (#9544) The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files. * Logging: New delay detector * Rename to SpanDurationLogger * Use Durations instead of nanoseconds * Use Durations instead of nanoseconds * refactor: Tracing (#9544) The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files. * Logging: New delay detector * refactor: Tracing (#9544) The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files. * Use Durations instead of nanoseconds
nikurt
added a commit
that referenced
this pull request
Sep 26, 2023
Continues #9540 Deletes the `DelayDetector` class and replaces the uses of it with spans. --------- Co-authored-by: Anton Puhach <anton@near.org> Co-authored-by: Simonas Kazlauskas <git@kazlauskas.me> Co-authored-by: Jakob Meier <mail@jakobmeier.ch>
nikurt
added a commit
that referenced
this pull request
Sep 26, 2023
* Logging: New delay detector * Logging: New delay detector * Logging: New delay detector * Logging: New delay detector * Rename to SpanDurationLogger * Use Durations instead of nanoseconds * refactor: Tracing (#9544) The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files. * Logging: New delay detector * Rename to SpanDurationLogger * Use Durations instead of nanoseconds * Use Durations instead of nanoseconds * refactor: Tracing (#9544) The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files. * Logging: New delay detector * refactor: Tracing (#9544) The `lib.rs` file accumulated structs and functionality from different areas. Split the file into several smaller files. * Use Durations instead of nanoseconds
nikurt
added a commit
that referenced
this pull request
Sep 26, 2023
Continues #9540 Deletes the `DelayDetector` class and replaces the uses of it with spans. --------- Co-authored-by: Anton Puhach <anton@near.org> Co-authored-by: Simonas Kazlauskas <git@kazlauskas.me> Co-authored-by: Jakob Meier <mail@jakobmeier.ch>
nikurt
added a commit
that referenced
this pull request
Oct 11, 2023
Continues #9540 Deletes the `DelayDetector` class and replaces the uses of it with spans. --------- Co-authored-by: Anton Puhach <anton@near.org> Co-authored-by: Simonas Kazlauskas <git@kazlauskas.me> Co-authored-by: Jakob Meier <mail@jakobmeier.ch>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Improves the logging system to automatically export durations of spans as metrics and detect spans taking too long.
This PR compliments the functionality Grafana Tempo.
Changes:
Performance impact should be minimal.
Memory usage increase is minimal.