LINPACK benchmarks

The LINPACK Benchmarks are a measure of a system's floating-point computing power. Introduced by Jack Dongarra, they measure how fast a computer solves a dense n by n system of linear equations Ax = b, which is a common task in engineering.

The latest version of these benchmarks is used to build the TOP500 list, ranking the world's most powerful supercomputers.^[1]

The aim is to approximate how fast a computer will perform when solving real problems. It is a simplification, since no single computational task can reflect the overall performance of a computer system. Nevertheless, the LINPACK benchmark performance can provide a good correction over the peak performance provided by the manufacturer. The peak performance is the maximal theoretical performance a computer can achieve, calculated as the machine's frequency, in cycles per second, times the number of operations per cycle it can perform. The actual performance will always be lower than the peak performance.^[2] The performance of a computer is a complex issue that depends on many interconnected variables. The performance measured by the LINPACK benchmark consists of the number of 64-bit floating-point operations, generally additions and multiplications, a computer can perform per second, also known as FLOPS. However, a computer's performance when running actual applications is likely to be far behind the maximal performance it achieves running the appropriate LINPACK benchmark.^[3]

The name of these benchmarks comes from the LINPACK package, a collection of algebra Fortran subroutines widely used in the 1980s, and initially tightly linked to the LINPACK benchmark. The LINPACK package has since been replaced by other libraries.

The LINPACK benchmark report appeared first in 1979 as an appendix to the LINPACK user's manual.^[4]

LINPACK was designed to help users estimate the time required by their systems to solve a problem using the LINPACK package, by extrapolating the performance results obtained by 23 different computers solving a matrix problem of size 100.

This matrix size was chosen due to memory and CPU limitations at that time:

• 10,000 floating-point entries from -1 to 1 are randomly generated to fill in a general, dense matrix,
• then, LU decomposition with partial pivoting is used for the timing.

Over the years, additional versions with different problem sizes, like matrices of order 300 and 1000, and constraints were released, allowing new optimization opportunities as hardware architectures started to implement matrix-vector and matrix-matrix operations.^[5]

Parallel processing was also introduced in the LINPACK Parallel benchmark in the late 1980s.^[2]

In 1991, the LINPACK was modified for^[6] solving problems of arbitrary size, enabling high performance computers (HPC) to get near to their asymptotic performance.

Two years later this benchmark was used for measuring the performance of the first TOP500 list.

LINPACK 100 is very similar to the original benchmark published in 1979 along with the LINPACK users' manual. The solution is obtained by Gaussian elimination with partial pivoting, with 2/3n³ + 2n² floating-point operations where n is 100, the order of the dense matrix A that defines the problem. Its small size and the lack of software flexibility doesn't allow most modern computers to reach their performance limits. However, it can still be useful to predict performances in numerically intensive user written code using compiler optimization.^[2]

LINPACK 1000 can provide a performance nearer to the machine's limit because in addition to offering a bigger problem size, a matrix of order 1000, changes in the algorithm are possible. The only constraints are that the relative accuracy can't be reduced and the number of operations will always be considered to be 2/3 n³ + 2 n², with n = 1000.^[2]

The previous benchmarks are not suitable for testing parallel computers,^[7] and the so-called Linpack's Highly Parallel Computing benchmark, or HPLinpack benchmark, was introduced. In HPLinpack the size n of the problem can be made as large as it is needed to optimize the performance results of the machine. Once again, 2/3n³ + 2n² will be taken as the operation count, with independence of the algorithm used. Use of the Strassen algorithm is not allowed because it distorts the real execution rate.^[8] The accuracy must be such that the following expression is satisfied:

${\displaystyle {\|Ax-b\|_{\infty } \over (\|A\|_{\infty }\|x\|_{\infty }+\|b\|_{\infty })n\epsilon }\leq O(1)}$,

where

• ${\displaystyle \epsilon }$ is the machine's precision, and
• n is the size of the problem,^[9]
• ${\displaystyle \|\cdot \|_{\infty }}$ is a matrix norm (largest row sum ^[10]) and
• ${\displaystyle O(1)}$ corresponds to the big-O notation.

For each computer system, the following quantities are reported:^[2]

• R_max: the performance in GFLOPS for the largest problem run on a machine.
• N_max: the size of the largest problem run on a machine.
• N_1/2: the size where half the Rmax execution rate is achieved.
• R_peak: the theoretical peak performance GFLOPS for the machine.

These results are used to compile the TOP500 list twice a year, with the world's most powerful computers.^[1] TOP500 measures these in double-precision floating-point format (FP64). The ratio of R_max to R_peak is called parallel efficiency or HPL efficiency.^[11] It is typically lower the more nodes a system has due to communication overhead. For example, a 1990s Cray Y-MP achieves about 90% HPL efficiency,^[12] while Frontier achieves about 70% in 2023.^[13]

The previous section describes the ground rules for the benchmarks. The actual implementation of the program can diverge, with some examples being available in Fortran,^[14] C ^[15] or Java.^[16]

HPL is a portable implementation of HPLinpack that was written in C, originally as a guideline, but that is now widely used to provide data for the TOP500 list, though other technologies and packages can be used. HPL generates a linear system of equations of order n and solves it using LU decomposition with partial row pivoting. It requires installed implementations of MPI and either BLAS or VSIPL to run.^[17]

Coarsely, the algorithm has the following characteristics:^[18][19]

• cyclic data distribution in 2D blocks
• LU factorization using the right-looking variant with various depths of look-ahead
• recursive panel factorization
• six different panel broadcasting variants
• bandwidth reducing swap-broadcast algorithm
• backward substitution with look-ahead of depth 1

The LINPACK benchmark is said to have succeeded because of the scalability^[20] of HPLinpack, the fact that it generates a single number, making the results easily comparable and the extensive historical data base it has associated.^[21] However, soon after its release, the LINPACK benchmark was criticized for providing performance levels "generally unobtainable by all but a very few programmers who tediously optimize their code for that machine and that machine alone",^[22] because it only tests the resolution of dense linear systems, which are not representative of all the operations usually performed in scientific computing.^[23] Jack Dongarra, the main driving force behind the LINPACK benchmarks, said that, while they only emphasize "peak" CPU speed and number of CPUs, not enough stress is given to local bandwidth and the network.^[24]

Thom Dunning, Jr., director of the National Center for Supercomputing Applications, had this to say about the LINPACK benchmark: "The Linpack benchmark is one of those interesting phenomena -- almost anyone who knows about it will deride its utility. They understand its limitations but it has mindshare because it's the one number we've all bought into over the years."^[25]

According to Dongarra, "the organizers of the Top500 are actively looking to expand the scope of the benchmark reporting" because "it is important to include more performance characteristic and signatures for a given system".^[26] One of the possibilities that is being considered to extend the benchmark for the TOP500 is the HPC Challenge Benchmark Suite.^[27] With the advent of petascale computers, traversed edges per second have started to emerge as a complementary metric to FLOPS measured by LINPACK. Another such metric is the HPCG benchmark, proposed by Dongarra.^[28]

According to Jack Dongarra, the running time required to obtain good performance results with HPLinpack is expected to increase. At a conference held in 2010, he said he expects running times of 2.5 days in "a few years".^[29]

• LAPACK

• TOP500 LINPACK
• a web-based LINPACK benchmark
• Intel Optimized LINPACK Benchmark