Note: Along with this month's post, I am excited to announce fantasyleaguereport.com. For the last year, I have been working on developing tools to make fantasy football more fun and with the public beta launch of Fantasy League Report I am making progress on that goal. Fantasy League Report is a web app and email service for ESPN fantasy football leagues. It automatically pulls your league's history so you can quickly compare how you match up with longtime league mates. Through weekly emails, Fantasy League Report assesses your abilities as a fantasy manager, supplying matchup recaps, coach grades, and league power rankings. So please give it a shot and reach out to me with any thoughts, comments, or bug reports. Now onto this month's post.
Ah the eternal debate.
Is fantasy football a game of luck or skill?
Everyone loves to argue that whenever things go their way it is based on skill, but anything that goes wrong is bad luck. That is cognitive bias at its best! But at the end of the day, fantasy football is a combination of luck and skill. I have previously tried to quantify this on a week-by-week, matchup level basis, concluding that luck may account for about 2 wins/losses for a team each year.
Since then I have been looking for a way to quantify luck vs skill from a more holistic level, including the draft, waivers, trades, and lineup decisions.
Enter a group of researchers from MIT who set about calculating the luck vs skill breakdown in daily fantasy sports. Starting from the assumption that if it was pure luck, you would win half of your games in the long run. So if players consistently and persistently (year after year) win significantly more or less than half their games (high or low skill), then that competition is skill-based. They use this logic to define how both fantasy and real sports align on the luck vs skill spectrum.
The above chart captures their metric (which they call R*) ranging from 0 (pure luck, literally flipping a coin) to 1 (pure skill), and overall it makes sense. For instance, compare the NHL vs the NBA. In Hockey, there are fewer games each year and few scoring opportunities per game, this leads to a lot of lucky wins and a game that is shifted to the left (more luck-driven). Meanwhile, in basketball, there are a lot of games and each game has lots of scoring opportunities. This means that most games end up being skill-based and you get a largely skill-based sport. (Note: the authors were surprised at the relatively high skill-based level of NFL, especially given the few games in a season and the few scoring opportunities, they leave it as a puzzle as I will for you).
The argument of the paper was that daily fantasy sports align pretty well with professional sports, with daily fantasy football coming in at roughly 55% skill, 45% luck (and therefore should not be considered gambling). In this post, I will be calculating the R* metric for season-long fantasy football and we can see where it falls on this chart.
The Data
We will be going back to our ESPN public league datasets and comparing the win rates of fantasy managers in 2019 and 2020. Starting from about 15,000 leagues in each year, I found the ~1500 leagues (representing ~12,000 teams) that were repeated across the 2 years. I then tried to control for managers who stopped paying attention by removing any team from the dataset that failed to set their lineup for 1 or more weeks in either 2019 or 2020. We are left with a sample of 4115 teams across 1252 leagues that played in 2019 and 2020.
We will first look at a simple correlation, between the points scored in 2019 vs points scored in 2020. This should remove the matchup luck which I have covered before and focus on if fantasy football players are able to consistently create a strong team. To allow for leagues with different scoring settings to be comparable, all points will be normalized as z-scores within the league each year. This means that we will be comparing how much better a team is compared to the other teams in their league for each year.
We will also calculate the R* value. I recommend going through the MIT paper for a detailed explanation, but the intuitive way to think about it is how well your 2019 win rate correlates with your 2020 win rate. (Note: it is slightly different than this, but this thought process will get you in the ballpark). So if fantasy football was pure skill, you would win the same amount of games year after year (good managers would win more than half their games, bad managers would win less than half their games). However, if the game was pure luck, then you would expect team win totals to be randomly distributed around .500 every year.
The Results
Points
Heat map of the z-scores of points scored in 2019 and 2020. The white line represents a linear regression based on the data. The positive slope suggests that teams who scored more than the league average in 2019 tended to do so again in 2020 (and vice versa). however, the R^2 value is only 0.01.
While we do get a significant correlation (p<0.001), that has more to do with the sample size. We are only explaining about 1% of the variation year to year here (with an R^2 value of 0.01) which basically means there is very little correlation between these two. This would suggest that that year-to-year success is driven by luck. (Note: I did look at correlating other metrics like wins, regular season standing, or final standing and all had worse correlations than points, which makes sense given that points are entirely under the manager's control while all the other factors are influenced by the other managers in the league).
R*
When we calculate the R* value we get 0.19. By this metric, that suggests that fantasy football in the season-long version is 20% skill and 80% luck, putting it in the ballpark of the stock market. Neither of these results is what I was expecting, with both being far further on the luck side compared to the skill side.
Limitations
Now I should say that by the standards of the MIT paper, my sample size is small. In the paper, they worked with anywhere from 50,000 (fantasy hockey) to 500,000 (fantasy football) players. Additionally, I am only comparing 2 years (2019 to 2020) and one of those years is a covid year! (thanks to u/ShinySuitTheory for reminding me to make this explicit), out of no where a wide receiver started at QB 2020 due to covid! Combined, that means there will likely be more noise in my dataset, which in turn will lead to this metric favoring luck more than skill. Additionally, many will argue that public fantasy football leagues are filled with less skilled fantasy players, which may be true, but I would argue that most of the less skilled fantasy players would have been filtered out of this dataset (and in fact if I loosen my dataset rules to allow for 1 or 2 missed weeks, I see the R* value drop).
The most likely explanation I think is that fantasy managers are competing within specific leagues and it is unlikely that fantasy skill level is evenly distributed across these leagues. (for those in multiple leagues, you can probably say which ones are full of fantasy sharks and which are not). So we could interpret the results as performance in YOUR SPECIFIC LEAGUE is based on 20% skill and 80% luck. Because fantasy managers will likely gravitate towards leagues where all managers are at a similar skill level (particularly when the league persists from year to year), this would trend the result towards the luck factor. For instance, if 2 teams of equal skill face-off, then the outcome should be pure luck. So if the entire league is full of managers of equal skill levels, then their win rates should trend towards .500 and a pure luck determination. So unlike the MIT data where managers played against random individuals or thousands of strangers at once, our data here consists of managers playing people they likely know (especially since the league had to stick around for 2 years to be considered).
Conclusions
Overall, I think this provides the base rate of luck vs skill in fantasy football, specifically that performance in any given league is 80% luck and 20% skill. Before you say that indicates that it is a game of pure luck, remember that this level of luck vs skill puts fantasy football in the same range as the stock market and people make millions of dollars off of their investing abilities.
Remember to check out fantasyleaguereport.com if you are in an ESPN league to get your league history.
Questions? Comments? Let me know at ac@alexcates.com. Want to read more breakdowns like this? sign up for my newsletter. Finally, like what I do? Consider supporting me on buy me a coffee.