Wednesday, January 14, 2009

Fantasy Sports Prognosticators Not So Accurate

The 2008 NFL regular season is now over, which also means that for the most part the fantasy football season is over. In a previous post, I mentioned that it would be nice to have some measure of accuracy of player projections.

Now this is not by any means determinative, but I took a few fantasy football magazines I had for this year, and compared their QB projected rankings to how they actually turned out. The stats were based on total performance (passing and rushing), not just TDs. The league I was in counted passing TDs for 3 pts instead of 6 pts, so QBs that maybe didn't pass as well but got some stats rushing may show up higher in this list than the rankings your league has. The ranking lists of the magazines also were based on a performance model, not just TDs, to keep the comparison consistent.

To be fair to the magazines, since I used as my sample what I had on hand, I omitted the actual names of the magazines and list them as A, B and C. The table below shows the fantasy QB rankings at the end of this year's regular season, along with the projected rankings from the 3 fantasy football magazines. Also, I averaged the ranking error for the 3 magazines under the "Variance" column.

Where the QB was not included (ranked too low) in a particular magazine's rankings, I insert an "N/A" and assume a ranking of 50. Magazine A had a longer list of QB rankings compared to magazines B and C.

Here's the results (table is at the end of the post):

  • Correct within 5 or less spots - 10 (31%)
  • Correct within 10 or less spots - 13 (40%)
  • Incorrect by 11 or more spots - 19 (59%)
  • Average ranking error - 15 spots
What is striking is how inaccurate the various fantasy magazines are with their rankings. It is interesting that these magazines, supposedly independent, are very similar with regard to their rankings. They by and large all predict the same performance. So what is the real difference between these various magazines?

On a whim, I wanted to see if the final QB stats ranking for the 2007 season was used, as is, for the 2008 QB stats ranking, how that would measure. For those QBs that ended up ranking too low for measurement in the final 2008 regular season stats, I assumed a ranking of 65. No changes were made to the 2008 projected rankings based on off-season changes. I took the final 2007 regular season rankings verbatim. In other words, Cleo Lemon was projected 25th, Josh McCown was projected 31st, etc. The results are:

  • Correct within 5 or less spots - 10 (31%)
  • Correct within 10 or less spots - 15 (46%)
  • Incorrect by 11 or more spots - 17 (53%)
  • Average ranking error - 17 spots
Pretty much the same. So, at least compared to the magazines considered, you could have simply taken last year's QB stats rankings as your cheat sheet and likely not done any worse. I didn't do the work, but from a cursory glance, taking the 2007 rankings and simply swapping QBs that took over from the previous year's starter, you likely would have been more accurate than the fantasy football magazines (at least the ones evaluated).

I did not look at the other positions, so it is possible QBs have a greater variation but that would have to be verified and I'll leave that to others to research. As of now, here's some evidence that fantasy football magazine projections (crafted by "experts") are not statistically more accurate than just saying what happened last year would happen this year.






































#QB NameMag AMag BMag CVariance
1Drew Brees5443
2Aaron Rodgers16191916
3Jay Cutler8885
4Philip Rivers18131311
5Kurt Warner37363631
6Peyton Manning2224
7Matt Cassel64N/AN/A48
8Donovan McNabb7662
9Tyler ThigpenN/AN/AN/A41
10David Garrard915153
11Chad Pennington27303018
12Tony Romo4339
13Eli Manning1312121
14Brett FavreN/AN/A4534
15Matt RyanN/A343424
16Jason Campbell1725246
17Kyle Orton31383919
18Joe Flacco33373818
19Ben Roethlisberger67712
20Jake Delhomme2114144
21Matt Schaub1217176
22Jeff Garcia2223231
23Trent Edwards2428284
24Kerry Collins50N/AN/A26
25Shaun Hill38404014
26JaMarcus Russell2622223
27Marc Bulger15111015
28Ryan Fitzpatrick66N/AN/A27
29Gus Frerotte45N/AN/A19
30Seneca Wallace53N/AN/A21
31Derek Anderson11101120
32Dan OrlovskyN/AN/AN/A18



Add to Technorati Favorites

No comments: