8000 [Benchmark] High compilation time variance on benchmark dashboards · Issue #152566 · pytorch/pytorch · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
[Benchmark] High compilation time variance on benchmark dashboards #152566
@huydhn

Description

@huydhn

The issue is reported by compiler team (@zou3519, @oulgen) in which the compilation time seems to have a higher variance across runs. This happens on both PT2 (no compiler cache) and CacheBench dashboard, which seems to indicate an underlying problem with the runner.

Image

Image

One potential explanation is that the H100/A100 where these benchmarks are running is multi-tenancy. So, other jobs running in parallel on the same runner could cause this.

cc @seemethere @malfet @pytorch/pytorch-dev-infra @ZainRizvi @clee2000

Metadata

Metadata

Assignees

Labels

module: ciRelated to continuous integrationmodule: infraRelates to CI infrastructuretriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    0