[Inference] FP8 dual gemm auto-tune and support compile parallelization #9151
Conversation
|
Thanks for your contribution! |
|
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #9151 +/- ##
========================================
Coverage 53.26% 53.26%
========================================
Files 652 652
Lines 105615 105615
========================================
Hits 56254 56254
Misses 49361 49361 ☔ View full report in Codecov by Sentry. |
c35ebb6 to
1934c3f
Compare
| @@ -0,0 +1,628 @@ | |||
| # Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved. | |||
There was a problem hiding this comment.
我建议在csrc/目录下建一个utils或者tools目录，把csrc/generate_code_dual_gemm_fused_kernels.py移进去，另外generate_code_dual_gemm_fused_kernels文件的命名也有点不太准确，改为auto_gen_fp8_fp8_dual_gemm_fused_kernels.py ?
There was a problem hiding this comment.
另外顺便把csrc/test_tune_cublaslt_gemm.py也移到utils或者tools目录下，test_tune_cublaslt_gemm命名改为tune_cublaslt_int8_gemm.py吧
There was a problem hiding this comment.
csrc/generate_code_gemm_fused_kernels.py文件同理～
PR types
New features
PR changes
Others
Description
添加FP8 cutlass dual gemm 自定义算子；
针对生成的cutlass算子的编译进行优化，在L20上将所有自定义算子编译时间由52分钟优化到3分钟，加速16.3倍；