This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.

High-Performance FP32 Matrix Multiplication on CPU

Step-by-step optimization of CUDA SGEMM

The repository targets the OpenCL gemm function performance optimization. It compares several libraries clBLAS, clBLAST, MIOpenGemm, Intel MKL(CPU) and cuBLAS(CUDA) on different matrix sizes/vendor's hardwares/OS. Out-of-the-box easy as MSVC, MinGW, Linux(CentOS) x86_64 binary provided. 在不同矩阵大小/硬件/操作系统下比较几个BLAS库的sgemm函数性能，提供binary，开盒即用。

maxas Scott Grey's maxas assembler sgemm explaining the (for me) missing parts https://github.com/NervanaSystems/maxas

GEMM

An alternative SGEMM implementation on AMD Vega Series

A benchmark framework for POWER and x86_64

a fast sgemm lib with fix 16 enable on arm 32

General Matrix Multiplication using NVIDIA Tensor Cores

CUDA Kernels on V100

CuPy first example computing GEMM with cuBlas, with handwritten cuda kernel and with NumPy-blas