The 2008 NFL regular season is now over, which also means that for the most part the fantasy football season is over. In a previous
post, I mentioned that it would be nice to have some measure of accuracy of player projections.
Now this is not by any means determinative, but I took a few fantasy football magazines I had for this year, and compared their QB projected rankings to how they actually turned out. The stats were based on total performance (passing and rushing), not just TDs. The league I was in counted passing TDs for 3 pts instead of 6 pts, so QBs that maybe didn't pass as well but got some stats rushing may show up higher in this list than the rankings your league has. The ranking lists of the magazines also were based on a performance model, not just TDs, to keep the comparison consistent.
To be fair to the magazines, since I used as my sample what I had on hand, I omitted the actual names of the magazines and list them as A, B and C. The table below shows the fantasy QB rankings at the end of this year's regular season, along with the projected rankings from the 3 fantasy football magazines. Also, I averaged the ranking error for the 3 magazines under the "Variance" column.
Where the QB was not included (ranked too low) in a particular magazine's rankings, I insert an "N/A" and assume a ranking of 50. Magazine A had a longer list of QB rankings compared to magazines B and C.
Here's the results (table is at the end of the post):
- Correct within 5 or less spots - 10 (31%)
- Correct within 10 or less spots - 13 (40%)
- Incorrect by 11 or more spots - 19 (59%)
- Average ranking error - 15 spots
What is striking is how inaccurate the various fantasy magazines are with their rankings. It is interesting that these magazines, supposedly independent, are very similar with regard to their rankings. They by and large all predict the same performance. So what is the real difference between these various magazines?
On a whim, I wanted to see if the final QB stats ranking for the 2007 season was used, as is, for the 2008 QB stats ranking, how that would measure. For those QBs that ended up ranking too low for measurement in the final 2008 regular season stats, I assumed a ranking of 65. No changes were made to the 2008 projected rankings based on off-season changes. I took the final 2007 regular season rankings verbatim. In other words, Cleo Lemon was projected 25th, Josh McCown was projected 31st, etc. The results are:
- Correct within 5 or less spots - 10 (31%)
- Correct within 10 or less spots - 15 (46%)
- Incorrect by 11 or more spots - 17 (53%)
- Average ranking error - 17 spots
Pretty much the same. So, at least compared to the magazines considered, you could have simply taken last year's QB stats rankings as your cheat sheet and likely not done any worse. I didn't do the work, but from a cursory glance, taking the 2007 rankings and simply swapping QBs that took over from the previous year's starter, you likely would have been more accurate than the fantasy football magazines (at least the ones evaluated).
I did not look at the other positions, so it is possible QBs have a greater variation but that would have to be verified and I'll leave that to others to research. As of now, here's some evidence that fantasy football magazine projections (crafted by "experts") are not statistically more accurate than just saying what happened last year would happen this year.
# | QB Name | Mag A | Mag B | Mag C | Variance |
1 | Drew Brees | 5 | 4 | 4 | 3 |
2 | Aaron Rodgers | 16 | 19 | 19 | 16 |
3 | Jay Cutler | 8 | 8 | 8 | 5 |
4 | Philip Rivers | 18 | 13 | 13 | 11 |
5 | Kurt Warner | 37 | 36 | 36 | 31 |
6 | Peyton Manning | 2 | 2 | 2 | 4 |
7 | Matt Cassel | 64 | N/A | N/A | 48 |
8 | Donovan McNabb | 7 | 6 | 6 | 2 |
9 | Tyler Thigpen | N/A | N/A | N/A | 41 |
10 | David Garrard | 9 | 15 | 15 | 3 |
11 | Chad Pennington | 27 | 30 | 30 | 18 |
12 | Tony Romo | 4 | 3 | 3 | 9 |
13 | Eli Manning | 13 | 12 | 12 | 1 |
14 | Brett Favre | N/A | N/A | 45 | 34 |
15 | Matt Ryan | N/A | 34 | 34 | 24 |
16 | Jason Campbell | 17 | 25 | 24 | 6 |
17 | Kyle Orton | 31 | 38 | 39 | 19 |
18 | Joe Flacco | 33 | 37 | 38 | 18 |
19 | Ben Roethlisberger | 6 | 7 | 7 | 12 |
20 | Jake Delhomme | 21 | 14 | 14 | 4 |
21 | Matt Schaub | 12 | 17 | 17 | 6 |
22 | Jeff Garcia | 22 | 23 | 23 | 1 |
23 | Trent Edwards | 24 | 28 | 28 | 4 |
24 | Kerry Collins | 50 | N/A | N/A | 26 |
25 | Shaun Hill | 38 | 40 | 40 | 14 |
26 | JaMarcus Russell | 26 | 22 | 22 | 3 |
27 | Marc Bulger | 15 | 11 | 10 | 15 |
28 | Ryan Fitzpatrick | 66 | N/A | N/A | 27 |
29 | Gus Frerotte | 45 | N/A | N/A | 19 |
30 | Seneca Wallace | 53 | N/A | N/A | 21 |
31 | Derek Anderson | 11 | 10 | 11 | 20 |
32 | Dan Orlovsky | N/A | N/A | N/A | 18 |