8000 debug: Add #[perf] to more actors by jakmeier · Pull Request #9551 · near/nearcore · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

debug: Add #[perf] to more actors #9551

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Sep 21, 2023
Merged

debug: Add #[perf] to more actors #9551

merged 2 commits into from
Sep 21, 2023

Conversation

jakmeier
Copy link
Contributor

This allows to observe more details about actor utilization when compiling with --features=performance_stats.

Running some loadtest with this reveals that ClientActor and ShardsManagerActor are the busiest in some cases, which would not be possible to observe without the instrumentation added in this commit.

@jakmeier
debug: Add #[perf] to more actors
98040c0 
This allows to observe more details about actor utilization when
compiling with `--features=performance_stats`.

Running some loadtest with this reveals that `ClientActor`
and `ShardsManagerActor` are the busiest in some cases, which
would not be possible to observe without the instrumentation added
in this commit.
@jakmeier jakmeier requested a review from nikurt September 21, 2023 11:21
@jakmeier jakmeier requested a review from a team as a code owner September 21, 2023 11:21
@jakmeier
Copy link
Contributor Author

I didn't even know this feature existed until this week :S

But it can give me output like this:

2023-09-20T16:56:31.166718Z  INFO near_performance_metrics::stats_enabled: Performance stats 15 threads (min ratio = 0.02)
2023-09-20T16:56:31.166801Z  WARN near_performance_metrics::stats_enabled:     ThreadId(50): ratio: 0.195 {"near_network::peer_manager::peer_manager_actor::PeerManagerActor", "run_later"}:ThreadId(34) C mem: 0 B
2023-09-20T16:56:31.166822Z  WARN near_performance_metrics::stats_enabled:         func ():0: cnt: 94 total: 971ms max: 149ms
2023-09-20T16:56:31.166827Z  WARN near_performance_metrics::stats_enabled:         func chain/network/src/peer_manager/peer_manager_actor.rs:715: cnt: 46 total: 3ms max: 0ms
2023-09-20T16:56:31.166838Z  WARN near_performance_metrics::stats_enabled:     ThreadId(46): ratio: 0.321 {"near_chain::blocking_io_actor::BlockingIoActor"}:ThreadId(34) C mem: 0 B
2023-09-20T16:56:31.166843Z  WARN near_performance_metrics::stats_enabled:         func ():0: cnt: 9 total: 1574ms max: 856ms
2023-09-20T16:56:31.166850Z  WARN near_performance_metrics::stats_enabled:     ThreadId(43): ratio: 0.407 {"near_chain::blocking_io_actor::BlockingIoActor"}:ThreadId(34) C mem: 0 B
2023-09-20T16:56:31.166852Z  WARN near_performance_metrics::stats_enabled:         func ():0: cnt: 13 total: 1500ms max: 532ms
2023-09-20T16:56:31.166858Z  WARN near_performance_metrics::stats_enabled:     ThreadId(45): ratio: 0.352 {"near_chain::blocking_io_actor::BlockingIoActor"}:ThreadId(34) C mem: 0 B
2023-09-20T16:56:31.166859Z  WARN near_performance_metrics::stats_enabled:         func ():0: cnt: 10 total: 1629ms max: 539ms
2023-09-20T16:56:31.166868Z  WARN near_performance_metrics::stats_enabled:     ThreadId(47): ratio: 0.392 {"near_chunks::shards_manager_actor::ShardsManagerActor", "run_later"}:ThreadId(34) C mem: 0 B
2023-09-20T16:56:31.166870Z  WARN near_performance_metrics::stats_enabled:         func ():0: cnt: 52 total: 1961ms max: 416ms
2023-09-20T16:56:31.166872Z  WARN near_performance_metrics::stats_enabled:         func chain/chunks/src/shards_manager_actor.rs:33: cnt: 9 total: 0ms max: 0ms
2023-09-20T16:56:31.166876Z  WARN near_performance_metrics::stats_enabled:     ThreadId(44): ratio: 0.283 {"near_chain::blocking_io_actor::BlockingIoActor"}:ThreadId(34) C mem: 0 B
2023-09-20T16:56:31.166879Z  WARN near_performance_metrics::stats_enabled:         func ():0: cnt: 19 total: 1184ms max: 422ms
2023-09-20T16:56:31.166883Z  WARN near_performance_metrics::stats_enabled:     ThreadId(42): ratio: 0.528 {"near_client::client_actor::ClientActor", "run_later"}:ThreadId(34) C mem: 0 B
2023-09-20T16:56:31.166886Z  WARN near_performance_metrics::stats_enabled:         func ():0: cnt: 95 total: 2631ms max: 844ms
2023-09-20T16:56:31.166888Z  WARN near_performance_metrics::stats_enabled:         func chain/client/src/client_actor.rs:1088: cnt: 31 total: 6ms max: 0ms
2023-09-20T16:56:31.166890Z  WARN near_performance_metrics::stats_enabled:         func chain/client/src/client_actor.rs:1514: cnt: 30 total: 2ms max: 0ms
2023-09-20T16:56:31.166901Z  WARN near_performance_metrics::stats_enabled:     ThreadId(73): ratio: 0.358 {"near_network::peer::peer_actor::PeerActor", "run_later"}:ThreadId(34) C mem: 0 B
2023-09-20T16:56:31.166906Z  WARN near_performance_metrics::stats_enabled:         func ():0: cnt: 77 total: 1787ms max: 272ms
2023-09-20T16:56:31.166910Z  WARN near_performance_metrics::stats_enabled:     ThreadId(62): ratio: 0.215 {"near_network::peer::peer_actor::PeerActor", "run_later"}:ThreadId(34) C mem: 0 B
2023-09-20T16:56:31.166912Z  WARN near_performance_metrics::stats_enabled:         func ():0: cnt: 77 total: 1074ms max: 138ms
2023-09-20T16:56:31.166916Z  INFO near_performance_metrics::stats_enabled:     Other threads ratio 0.000
2023-09-20T16:56:31.166918Z  INFO near_performance_metrics::stats_enabled: Total ratio = 3.050
2023-09-20T16:56:31.166921Z  INFO near_performance_metrics::stats_enabled: Futures waiting for completion
2023-09-20T16:56:31.166926Z  INFO near_performance_metrics::stats_enabled:     future chain/chunks/src/shards_manager_actor.rs:33 1
2023-09-20T16:56:31.166928Z  INFO near_performance_metrics::stats_enabled:     future chain/client/src/client_actor.rs:1088 1
2023-09-20T16:56:31.166929Z  INFO near_performance_metrics::stats_enabled:     future chain/network/src/peer_manager/peer_manager_actor.rs:621 1
2023-09-20T16:56:31.166931Z  INFO near_performance_metrics::stats_enabled:     future chain/network/src/peer_manager/peer_manager_actor.rs:715 1
2023-09-20T16:56:31.166932Z  INFO near_performance_metrics::stats_enabled:     future chain/client/src/client_actor.rs:1514 1
2023-09-20T16:56:31.166933Z  INFO near_performance_metrics::stats_enabled:     future chain/network/src/peer_manager/peer_manager_actor.rs:375 1

Notes:

  • Usually this is printed once per minute, I changed it to once per 5s for this. Therefore, 3000ms means a particular thread was active 60% of the time.
  • BlockingIoActor is from my experimental branch trying out some stuff, not something that exists today on master. It uses a sync arbiter, which means each instance has it's own OS-level thread, unlike normal arbiters that share one thread for all actors through an event loop.

nikurt
nikurt approved these changes Sep 21, 2023
View reviewed changes
Copy link
Contributor
@nikurt nikurt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, I love tools like that!

@jakmeier jakmeier enabled auto-merge September 21, 2023 11:44
@jakmeier jakmeier added this pull request to the merge queue Sep 21, 2023
Merged via the queue into near:master with commit 6bf8da3 Sep 21, 2023
@jakmeier jakmeier deleted the perf_on_actors branch September 21, 2023 12:30
nikurt pushed a commit that referenced this pull request Sep 22, 2023
@jakmeier
debug: Add #[perf] to more actors (#9551)
6fbf4ce 
* debug: Add #[perf] to more actors

This allows to observe more details about actor utilization when
compiling with `--features=performance_stats`.

Running some loadtest with this reveals that `ClientActor`
and `ShardsManagerActor` are the busiest in some cases, which
would not be possible to observe without the instrumentation added
in this commit.

* cargo fmt
nikurt pushed a commit that referenced this pull request Sep 26, 2023
@jakmeier
debug: Add #[perf] to more actors (#9551)
3ef33d7 
* debug: Add #[perf] to more actors

This allows to observe more details about actor utilization when
compiling with `--features=performance_stats`.

Running some loadtest with this reveals that `ClientActor`
and `ShardsManagerActor` are the busiest in some cases, which
would not be possible to observe without the instrumentation added
in this commit.

* cargo fmt
nikurt pushed a commit that referenced this pull request Sep 26, 2023
@jakmeier
debug: Add #[perf] to more actors (#9551)
3bb3286 
* debug: Add #[perf] to more actors

This allows to observe more details about actor utilization when
compiling with `--features=performance_stats`.

Running some loadtest with this reveals that `ClientActor`
and `ShardsManagerActor` are the busiest in some cases, which
would not be possible to observe without the instrumentation added
in this commit.

* cargo fmt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nikurt nikurt nikurt approved these changes

Assignees
No one assigned
Labels
None yet
45C9
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jakmeier @nikurt
0