Every week in the fall, thousands of people play fantasy football, making millions of decisions about who to start. ESPN (and all other sites that host fantasy football leagues) will provide projections to help. Every Sunday when I look at the projections I am tempted just to follow them. But how good are the projections? A quick search will find people complaining about how the projections were wrong for this player and that player. But are they actually that bad? In this post, I aim to figure out how good ESPN's projections are.
Player Projections
To figure this out, I pulled the weekly projections and actual scores from 2019 for every player rostered in a 10 team standard scoring league plus the top 50 free agents at each position. We can then compare both the raw and percent differences across all players and see how ESPN does. For the purposes of this, I removed any player who was projected for less than 1 point and I've narrowed the scope to be within 10 points of the projection or within 150% (both captures the majority of the data). Both graphs are available in the slideshow below.
Violin plots showing the distribution of the point differences. The white dot represents the mean, while the thicker black box represents the interquartile range.
At first blush, it looks ok but not great for ESPN. While they are only off by 2-3 points for most of the players, they seem to consistently overshoot on a player's projection by about 2 points or about 50%. Now that 50% seems like a lot given the raw difference which to me suggests that a lot of the lower end players (players projected for around 5 points) are driving this effect. So lets factor that in. Below is a graph of boxplots of projected vs actual points. This time, we bin players based on what they were projected for. I've also added in a diagonal line which represents if ESPN was 100% accurate and counts at the top of how many players were projected for that total.
Box plots of how many points a player scored, grouped by how many points they were projected for.
As expected, a lot of the players are in the lower projections and tend to underperform just a little. However the underperforming still holds for a lot of projected totals, with being projected for 11, 13, and 17 jumping out to me as prime projections to underperform. That said, ESPN's projections look pretty good overall here, particularly in the 8-20 range (excluding the 11, 13, and 17) which represents the majority of players you would be starting.
Let's take this one step further and group by position. ESPN may be really good a predicting running back performance but not kickers? We can create the same graph as above but group by position. The slideshow below contains the graphs for each position
Box plots of how many points a player scored, grouped by how many points they were projected for. Each graph contains only players from a different position.
A couple of interesting notes. First, ESPN consistently seems to underpredict quarterbacks and kickers (as represented by the fact that the box plots are above the perfect accuracy line). Additionally, while ESPN tends to do a good job with the other positions, we see the 11 point error to be still visible in both running backs and wide receivers. I do not know what to make of this except to be wary of players projected for 11 points. Finally, as a word of caution when looking at these graphs, be mindful of the total player counts at the top. For instance, a projection of 19 points looks really bad for running backs, but it only happened 3 times in 2019, not enough to draw any meaningful conclusions.
One last question about player projections is does ESPN's projections get better with time? I would expect there to be more variance and error early in the year when we are still learning how all the teams look and would get better as we learn which defenses are good, which players made a leap, which older players are dropping off. However, looking at the data that is not really what we see
Box plots of the raw difference between projected point total and actual point total for all players, grouped by week.
Team Projections
Now that we have dissected the player projections, let's look at the team level. End of the day, it is how the team does as a whole that matters. We can do some the same comparison to see how accurate ESPN is on the team level. In this instance, we will need a lot of teams, so I pulled every team from 500 different public leagues on ESPN, resulting in almost 60,000 data points. We can then take their projected and actual points at the team level and create our distributions.
Violin plots showing the distribution of the point differences at the team level. The white dot represents the mean, while the thicker black box represents the interquartile range.
At the team level, 2 interesting things come out. First is that on average espn's projections at the team level are really accurate (mean is -1.7% or off by about 2 points. At the same time, the deviation is huge here. The distributions shown are really broad so while on average ESPN is accurate at the team level, there is a lot of variation on any given week.
Next, we can look at projections based on point level. With the larger number of possible values, we will just do a line plot with the shaded area representing a 95% confidence interval. Below are all the scores and a zoomed-in version of scores from 70-130, which tends to be the scores you see in a game. In both cases, the red line represents the scores if ESPN was 100% accurate.
Line graph of projected vs actual points at the team level, shading represents a 95% confidence interval while the red line represents while the red line would be perfect predictions
As before, ESPN's team level projections tend to be pretty good on average. There is a slight dip around 110 projected points, but not much. There is also a little noise around the edges, partly due to fewer teams making up the estimate, but in general, you can trust ESPN's projections.
Finally, as with player projections, maybe ESPN gets better with time in its projections? Overall again, it doesn't look like ESPN got better with time, but there is an interesting seesaw at the beginning of the season where they overestimate, then underestimate, then back.
Box plots of raw error of team projections, grouped by week
Conclusion
So what does this all mean for your Fantasy Football team? First and foremost, know that on the whole, ESPN's projections are pretty good. Second, Kickers and QBs will tend to score a little more than projected. However because this is across the board, it ends up being a wash as your opponents at those positions will tend to outscore their projections too. Third, maybe avoid players projected for 11 points, that is by far the most interesting mystery here and I don't know what to make of that, but I will be cautious of them going forward. Finally, do take ESPN's team projections seriously, on average the error at the player level tends to even out so if you are projected to lose definitely look for higher variance players.
Questions? Comments? Let me know at ac@alexcates.com. Want to read more breakdowns like this? sign up for my newsletter here. Finally, like what I do? Consider supporting me on buy me a coffee.