10000 Add python stack tracing option on on-demand flow by slgong-fb · Pull Request #628 · pytorch/kineto · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add python stack tracing option on on-demand flow #628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Add python stack tracing option on on-demand flow #628

wants to merge 1 commit into from

Conversation

slgong-fb
Copy link
Contributor

Summary:
Changes:

  1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
  2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
  3. abstract class also changed.

Differential Revision: D37410204

@facebook-github-bot
Copy link
Contributor
8000

This pull request was exported from Phabricator. Differential Revision: D37410204

slgong-fb added a commit to slgong-fb/kineto that referenced this pull request Jul 6, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
22d70f6 
Summary:
X-link: pytorch/pytorch#80919

Pull Request resolved: pytorch#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Differential Revision: D37410204

fbshipit-source-id: b7ae9b49d87e58b84123c852ba79fe8e80aecbaf
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37410204

slgong-fb added a commit to slgong-fb/kineto that referenced this pull request Jul 6, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
4efd2d8 
Summary:
X-link: pytorch/pytorch#80919

Pull Request resolved: pytorch#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Differential Revision: D37410204

fbshipit-source-id: 67061846905eeb44aa38f70654731acfde7aa8ad
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37410204

slgong-fb added a commit to slgong-fb/kineto that referenced this pull request Jul 6, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
fe11558 
Summary:
X-link: pytorch/pytorch#80919

Pull Request resolved: pytorch#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Differential Revision: D37410204

fbshipit-source-id: f3f98c161897d39e57ce1c3df55d6d69ce8c0214
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37410204

slgong-fb added a commit to slgong-fb/kineto that referenced this pull request Jul 11, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
abf24b6 
Summary:
X-link: pytorch/pytorch#80919

Pull Request resolved: pytorch#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Trace after changes by running //kineto/libkineto/fb/integration_tests/trace_tester.cpp (requested by chaekit)
https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2F0%2F1657304871%2F127.0.0.1%2Flibkineto_activities_3502962.json.gz&bucket=gpu_traces

Reviewed By: chaekit

Differential Revision: D37410204

fbshipit-source-id: b79f0771461b3e67ca3fa4914d0be0072e81d003
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37410204

slgong-fb added a commit to slgong-fb/kineto that referenced this pull request Jul 12, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
4eed8d0 
Summary:
X-link: pytorch/pytorch#80919

Pull Request resolved: pytorch#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Trace after changes by running //kineto/libkineto/fb/integration_tests/trace_tester.cpp (requested by chaekit)
https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2F0%2F1657304871%2F127.0.0.1%2Flibkineto_activities_3502962.json.gz&bucket=gpu_traces

Reviewed By: chaekit

Differential Revision: D37410204

fbshipit-source-id: 094d872f33cb19a56fe92e06918bfd3b0410b210
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37410204

8000
slgong-fb added a commit to slgong-fb/kineto that referenced this pull request Jul 14, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
c9412bd 
Summary:
X-link: pytorch/pytorch#80919

Pull Request resolved: pytorch#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Trace after changes by running //kineto/libkineto/fb/integration_tests/trace_tester.cpp (requested by chaekit)
https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2F0%2F1657304871%2F127.0.0.1%2Flibkineto_activities_3502962.json.gz&bucket=gpu_traces

Reviewed By: chaekit

Differential Revision: D37410204

fbshipit-source-id: 093faffb6e3b14472ead9f84db9b1472d83885ad
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37410204

slgong-fb added a commit to slgong-fb/kineto that referenced this pull request Jul 18, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
a0e129f 
Summary:
X-link: pytorch/pytorch#80919

Pull Request resolved: pytorch#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Trace after changes by running //kineto/libkineto/fb/integration_tests/trace_tester.cpp (requested by chaekit)
https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2F0%2F1657304871%2F127.0.0.1%2Flibkineto_activities_3502962.json.gz&bucket=gpu_traces

Reviewed By: chaekit

Differential Revision: D37410204

fbshipit-source-id: 7e5b705fba676406a42a721fab36768db03ce9f9
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37410204

slgong-fb added a commit to slgong-fb/kineto that referenced this pull request Jul 19, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
a165ea7 
Summary:
X-link: pytorch/pytorch#80919

Pull Request resolved: pytorch#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Trace after changes by running //kineto/libkineto/fb/integration_tests/trace_tester.cpp (requested by chaekit)
https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2F0%2F1657304871%2F127.0.0.1%2Flibkineto_activities_3502962.json.gz&bucket=gpu_traces

Reviewed By: chaekit

Differential Revision: D37410204

fbshipit-source-id: 569c05335ca47701bd0c5a6e821b986846766702
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37410204

@slgong-fb
Copy link
Contributor Author
slgong-fb commented Jul 19, 2022 •
edited
Loading

CI test failures have the same issue discussed in #631

@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
f03b522 
Summary:
X-link: pytorch/pytorch#80919

Pull Request resolved: pytorch#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Trace after changes by running //kineto/libkineto/fb/integration_tests/trace_tester.cpp (requested by chaekit)
https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2F0%2F1657304871%2F127.0.0.1%2Flibkineto_activities_3502962.json.gz&bucket=gpu_traces

Reviewed By: chaekit

Differential Revision: D37410204

fbshipit-source-id: c9d38d0dd925e54153c95ee2322feaa2ed0fa486
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37410204

slgong-fb added a commit to slgong-fb/pytorch that referenced this pull request Jul 22, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (pytorch#80919)
7f411f3 
Summary:
Pull Request resolved: pytorch#80919

X-link: pytorch/kineto#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Trace after changes by running //kineto/libkineto/fb/integration_tests/trace_tester.cpp (requested by chaekit)
https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2F0%2F1657304871%2F127.0.0.1%2Flibkineto_activities_3502962.json.gz&bucket=gpu_traces

Test Plan:
1. launch a python test case with the following command for on-demand flow:
echo -e "PYTHON_STACK_TRACE=true" > /tmp/scott_kineto.conf && dyno gputrace --gputrace_duration 300ms --gpuconf /tmp/scott_kineto.conf

2. Then, we can see with_stack enabled as intended from output log:
INFO:2022-06-27 15:00:16 1009443:1011716 kineto_client_interface.cpp:22] withStack : 1

Reviewed By: chaekit

Differential Revision: D37410204

fbshipit-source-id: 7962c05b3bccb1359dfc46d832abda10caeb0e81
facebook-github-bot pushed a commit to pytorch/pytorch that referenced this pull request Jul 22, 2022
@slgong-fb @facebook-github-bot
Add python stack tracing option on on-demand flow (#80919)
b9a3e63 
Summary:
Pull Request resolved: #80919

X-link: pytorch/kineto#628

Changes:
1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Trace after changes by running //kineto/libkineto/fb/integration_tests/trace_tester.cpp (requested by chaekit)
https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree%2Ftraces%2Fdynocli%2F0%2F1657304871%2F127.0.0.1%2Flibkineto_activities_3502962.json.gz&bucket=gpu_traces

Test Plan:
1. launch a python test case with the following command for on-demand flow:
echo -e "PYTHON_STACK_TRACE=true" > /tmp/scott_kineto.conf && dyno gputrace --gputrace_duration 300ms --gpuconf /tmp/scott_kineto.conf

2. Then, we can see with_stack enabled as intended from output log:
INFO:2022-06-27 15:00:16 1009443:1011716 kineto_client_interface.cpp:22] withStack : 1

Reviewed By: chaekit

Differential Revision: D37410204

fbshipit-source-id: 2df900afc57ddab776b7cf1fd303debd9eafe8a1
@robieta
Copy link
robieta commented Jul 22, 2022

@slgong-fb This is a BC breaking change and is preventing us from updating the pinned version of Kineto in PyTorch. Can we revert?

@robieta
Copy link
robieta commented Jul 22, 2022

Specifically:

/var/lib/jenkins/workspace/torch/csrc/profiler/kineto_client_interface.cpp:21:8:error: ‘void torch::profiler::impl::{anonymous}::LibKinetoClient::start()’ marked ‘override’, but does not override
   void start() override {
^~~~~

pytorch/pytorch#80974

@robieta robieta mentioned this pull request Jul 22, 2022
Closed
@facebook-github-bot
Copy link
Contributor

This pull request has been reverted by aab809f.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
cla signed fb-exported Reverted
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@slgong-fb @facebook-github-bot @robieta
0