More Web Proxy on the site http://driver.im/

Forecast 2003

The results

This article will present the results of the Forecast experiment. New readers are invited to read the concept behind the Forecast series before continuing.

A special thanks to Sylvain Cognet for offering to compile, and compiling, the 2003 OPS and ERA for the 32 players in this study. (I love Primer readers! I sometimes get these surprises, where a dedicated reader will just compile useful data, and send it along. This was certainly a time-saver for me.)

This table presents each player's:

Baseline Forecast: average of last three years, with an age adjustment

Systematic Forecasters: average of the 6 forecasting engines (Palmer, Shandler, Silver, Szymborski, Tippett, Warren)

Primer Readers: average of the 165 ballots from the Primer readers

2003: actual results

Next 3 columns (Absolute): absolute difference between the picks and actual, after adjusting for league average (e.g, a pick of .830 with a league expectation of .730, against an actual of .840 with a league average of .740 is a perfect pick, giving a difference of .000)

Last 3 columns (Relative): taking the 3 preceding columns, but comparing to the minimum differential among the 3 picks (e.g., if the differentials were .060, .050, .020 in the preceding 3 columns, then the values in these last 3 columns would be .040, .030, .000).

Absolute Relative
Player Baseline Forecasters Primer Readers 2003 Baseline Forecasters Primer Readers Baseline Forecasters Primer Readers
Barry Bonds 1.231 1.248 1.268 1.278 0.052 0.025 0.006 0.046 0.019 0
Jim Thome 0.979 1.03 1.019 0.958 0.016 0.077 0.065 0 0.061 0.049
Gary Sheffield 0.949 0.952 0.939 1.023 0.079 0.066 0.08 0.013 0 0.014
Troy Glaus 0.949 0.883 0.887 0.807 0.137 0.081 0.084 0.056 0 0.003
Luis Gonzalez 0.934 0.931 0.871 0.934 0.005 0.002 0.059 0.003 0 0.057
J.D. Drew 0.895 0.865 0.873 0.886 0.004 0.016 0.009 0 0.012 0.005
Pat Burrell 0.895 0.892 0.956 0.713 0.177 0.184 0.247 0 0.007 0.07
Moises Alou 0.869 0.816 0.816 0.819 0.045 0.002 0.001 0.044 0.001 0
Richard Hidalgo 0.858 0.821 0.798 0.957 0.104 0.131 0.155 0 0.027 0.051
Jeremy Giambi 0.84 0.872 0.898 0.696 0.139 0.181 0.206 0 0.042 0.067
Sean Casey 0.808 0.786 0.766 0.758 0.045 0.033 0.012 0.033 0.021 0
Roberto Alomar 0.797 0.776 0.793 0.682 0.11 0.099 0.115 0.011 0 0.016
Jacque Jones 0.795 0.822 0.814 0.797 0.007 0.03 0.021 0 0.023 0.014
Torii Hunter 0.79 0.837 0.843 0.763 0.022 0.079 0.084 0 0.057 0.062
Rich Aurilia 0.773 0.79 0.798 0.735 0.033 0.06 0.067 0 0.027 0.034
Rondell White 0.77 0.726 0.776 0.829 0.064 0.098 0.049 0.015 0.049 0
Adam Kennedy 0.766 0.74 0.75 0.743 0.018 0.002 0.011 0.016 0 0.009
Jeromy Burnitz 0.741 0.737 0.724 0.786 0.05 0.044 0.058 0.006 0 0.014
Jeff Cirillo 0.74 0.689 0.723 0.555 0.18 0.139 0.172 0.041 0 0.033
Jose Hernandez 0.717 0.831 0.836 0.634 0.078 0.202 0.206 0 0.124 0.128
Marquis Grissom 0.673 0.717 0.692 0.79 0.122 0.068 0.094 0.054 0 0.026

Freddy Garcia 3.41 3.807 3.86 4.51 1.2 0.63 0.57 0.63 0.06 0
Javier Vazquez 3.41 3.665 3.61 3.24 0.07 0.5 0.45 0 0.43 0.38
Ryan Dempster 4.19 4.62 4.56 6.54 2.45 1.84 1.9 0.61 0 0.06
Kip Wells 4.32 4.227 4.1 3.28 0.94 1.03 0.9 0.04 0.13 0
Chan Ho Park 4.59 4.697 4.64 7.58 3.09 2.81 2.86 0.28 0 0.05
Matt Clement 4.6 3.843 3.95 4.11 0.39 0.19 0.08 0.31 0.11 0
Jamey Wright 4.76 4.887 5.1 4.26 0.4 0.7 0.92 0 0.3 0.52
Aaron Sele 4.77 4.647 4.83 5.77 1.1 1.05 0.86 0.24 0.19 0
Shawn Estes 4.91 4.337 4.79 5.73 0.92 1.32 0.86 0.06 0.46 0
Kenny Rogers 5.35 4.41 4.52 4.57 0.68 0.08 0.03 0.65 0.05 0
Todd Ritchie 5.62 4.768 5.16 5.08 0.44 0.23 0.16 0.28 0.07 0
League OPS 0.76 0.75 0.751 0.755 0.088 0.093 0.108 0.016 0.022 0.031
League ERA 4.49 4.312 4.31 4.39 1.33 1.15 1.16 0.28 0.16 0.09

The last 2 line needs a little explanation. The results under the "absolute" columns are the standard deviations of the values in those columns. The results under the "relative" columns are the average of the values in those columns.

Interpreting the Absolute differences

Jumping straight to the standard deviations, we see that the Baseline forecast did the best job with the hitters and worst with the pitchers. The Primer readers did the worst with the hitters. The Systematic Forecasters just beat out the Primer readers for pitchers.

You can go through the individual picks in these 3 columns, but I find it more interesting to look at...

Interpreting the Relative differences (Pitchers)

We see here that the Primer readers just nailed the pitcher forecasts. Of the 11 pitchers, the Primer readers had the closest pick in 7 of them. In 2 others, they were within .10 runs from the closest forecast pick. Primer readers blew it on Javy Vazquez and Jamey Wright.

How did the Systematic Forecasters do with their vaunted research and engine? They picked 2 pitchers better than the other forecasters. Another 6 were within .20 runs. They blew it on 3 pitchers, including Javy Vazquez.

And the Baseline, the thought-free system? They were closest on the 2 pitchers that the Primer readers misread the most (Vazquez and Wright). They had another 2 within .20 runs. And the rest were just bad. The 3 worst picks out of the whole group were all from the Baseline: Freddy Garcia, Ryan Dempster, and Kenny Rogers.

There is a certain amount of information about a pitcher that is not captured in the stats, especially ERA. The Readers and Forecaster knew that there was something a bit more special about Clement that wasn't captured, for example, and they made their picks accordingly. Javy Vazquez, a pitcher near and dear to me, for some reason did not elicit the same faith from the Readers or Forecasters.

All in all, Primer readers did a great job in forecasting the pitchers.

On to the Hitters

It seems that the amount of intuition required with pitchers is not at all required with hitters. The lesson here is: trust the numbers.

For best picks, the Baseline nailed 9 of the 21, the Systematic Forecasters got 8 of them, and the Primer readers just 4. If we look at the blown picks (difference of greater than .050), the Primer readers blew it on 6 hitters (Gonzalez, Burrell, Hidalgo, Giambi, Hunter, and Hernandez), while the Baseline blew it on 2 (Glaus, Grissom), and the Forecasters on 3 (Hernandez, Thome, and Hunter).

Overall, the Baseline did the best job with the hitters, both in terms of the absolute and relative differences, and the Primer readers the worst.

Recap

The 32 players were chosen because they had at least 3 years of performance to analyze, and that their year-to-year performance was very inconsistent (whether by luck, design, or injury). The test in this experiment was to see if the Systematic Forecasters would be able to interpret the numbers in a special way, or if the Primer reader's gut feelings would see something extra. Among the hitters, both of these groups failed to see anything beyond what the numbers said. In fact, they saw things that just weren't there. Among pitchers, the peripherals found by the Forecasters, or the "stuff" established by the Primer readers was enough to thoroughly beat the Baseline.

Conclusion? Trust the numbers you see (which is enough for hitters), and fill-in the information missing (which is the case with pitchers, be it health or mechanics). Any extra nuance that you find just doesn't have the impact you'd hope. None of the 3 groups dominated the others. This was about as close to a draw as you'd expect.

Next week, I'll look at the picks of 165 Primer readers, and crown a champion. I'll also break down the picks of the 6 Systematic Forecasters.