Fantasy Sports Prognosticators Not So Accurate

The 2008 NFL regular season is now over, which also means that for the most part the fantasy football season is over. In a previous post, I mentioned that it would be nice to have some measure of accuracy of player projections.

Now this is not by any means determinative, but I took a few fantasy football magazines I had for this year, and compared their QB projected rankings to how they actually turned out. The stats were based on total performance (passing and rushing), not just TDs. The league I was in counted passing TDs for 3 pts instead of 6 pts, so QBs that maybe didn't pass as well but got some stats rushing may show up higher in this list than the rankings your league has. The ranking lists of the magazines also were based on a performance model, not just TDs, to keep the comparison consistent.

To be fair to the magazines, since I used as my sample what I had on hand, I omitted the actual names of the magazines and list them as A, B and C. The table below shows the fantasy QB rankings at the end of this year's regular season, along with the projected rankings from the 3 fantasy football magazines. Also, I averaged the ranking error for the 3 magazines under the "Variance" column.

Where the QB was not included (ranked too low) in a particular magazine's rankings, I insert an "N/A" and assume a ranking of 50. Magazine A had a longer list of QB rankings compared to magazines B and C.

Here's the results (table is at the end of the post):

• Correct within 5 or less spots - 10 (31%)
• Correct within 10 or less spots - 13 (40%)
• Incorrect by 11 or more spots - 19 (59%)
• Average ranking error - 15 spots

What is striking is how inaccurate the various fantasy magazines are with their rankings. It is interesting that these magazines, supposedly independent, are very similar with regard to their rankings. They by and large all predict the same performance. So what is the real difference between these various magazines?

On a whim, I wanted to see if the final QB stats ranking for the 2007 season was used, as is, for the 2008 QB stats ranking, how that would measure. For those QBs that ended up ranking too low for measurement in the final 2008 regular season stats, I assumed a ranking of 65. No changes were made to the 2008 projected rankings based on off-season changes. I took the final 2007 regular season rankings verbatim. In other words, Cleo Lemon was projected 25th, Josh McCown was projected 31st, etc. The results are:

• Correct within 5 or less spots - 10 (31%)
• Correct within 10 or less spots - 15 (46%)
• Incorrect by 11 or more spots - 17 (53%)
• Average ranking error - 17 spots

Pretty much the same. So, at least compared to the magazines considered, you could have simply taken last year's QB stats rankings as your cheat sheet and likely not done any worse. I didn't do the work, but from a cursory glance, taking the 2007 rankings and simply swapping QBs that took over from the previous year's starter, you likely would have been more accurate than the fantasy football magazines (at least the ones evaluated).

I did not look at the other positions, so it is possible QBs have a greater variation but that would have to be verified and I'll leave that to others to research. As of now, here's some evidence that fantasy football magazine projections (crafted by "experts") are not statistically more accurate than just saying what happened last year would happen this year.

#	QB Name	Mag A	Mag B	Mag C	Variance
1	Drew Brees	5	4	4	3
2	Aaron Rodgers	16	19	19	16
3	Jay Cutler	8	8	8	5
4	Philip Rivers	18	13	13	11
5	Kurt Warner	37	36	36	31
6	Peyton Manning	2	2	2	4
7	Matt Cassel	64	N/A	N/A	48
8	Donovan McNabb	7	6	6	2
9	Tyler Thigpen	N/A	N/A	N/A	41
10	David Garrard	9	15	15	3
11	Chad Pennington	27	30	30	18
12	Tony Romo	4	3	3	9
13	Eli Manning	13	12	12	1
14	Brett Favre	N/A	N/A	45	34
15	Matt Ryan	N/A	34	34	24
16	Jason Campbell	17	25	24	6
17	Kyle Orton	31	38	39	19
18	Joe Flacco	33	37	38	18
19	Ben Roethlisberger	6	7	7	12
20	Jake Delhomme	21	14	14	4
21	Matt Schaub	12	17	17	6
22	Jeff Garcia	22	23	23	1
23	Trent Edwards	24	28	28	4
24	Kerry Collins	50	N/A	N/A	26
25	Shaun Hill	38	40	40	14
26	JaMarcus Russell	26	22	22	3
27	Marc Bulger	15	11	10	15
28	Ryan Fitzpatrick	66	N/A	N/A	27
29	Gus Frerotte	45	N/A	N/A	19
30	Seneca Wallace	53	N/A	N/A	21
31	Derek Anderson	11	10	11	20
32	Dan Orlovsky	N/A	N/A	N/A	18