Fidelity metrics for estimation models

Estimation models play a vital role in many aspects of day to day life. Extremely complex estimation models are employed in the design space exploration of SoCs, and the efficacy of these estimation models is usually measured by the absolute error of the models compared to known actual results. Such absolute error based metrics can often result in over-designed estimation models, with a number of researchers suggesting that fidelity of an estimation model (correlation between the ordering of the estimated points and the ordering of the actual points) should be examined instead of, or in addition to, the absolute error.

In this paper, for the first time, we propose four metrics to measure the fidelity of an estimation model, in particular for use in design space exploration. The first two are based on two well known rank correlation coefficients. The other two are weighted versions of the first two metrics, to give importance to points nearer the Pareto front. The proposed fidelity metrics range from -1 to 1, where a value of 1 reflects a perfect positive correlation while a value of -1 reflects a perfect negative correlation. The proposed fidelity metrics were calculated for a single processor estimation model and a multiprocessor estimation model to observe their behavior, and were compared against the models' absolute error. For the multiprocessor estimation model, even though the worst average and maximum absolute error of 6.40% and 16.61% respectively can be considered reasonable in design automation, the worst fidelity of 0.753 suggests that the multiprocessor estimation model may not be as good a model (compared to an estimation model with same or higher absolute errors but a fidelity of 0.95) as depicted by its absolute accuracy, leading to an over-designed estimation model.