Ah, the start sit article. It is a staple of any fantasy football coverage. Here are the 3 players you want to make sure you start this week, here are the 3 to avoid. There is something so tantalizing and simple about the article. But here is the thing, are they useful? Or are they just a fun excuse to talk about things everyone knows?
To test this, I will be using the start sit records collected by reddit users u/joshmich88 and u/fazik93 who have been posting consolidated start em sit ems for the past 2 years, though I will focus on the performance of these articles in 2019.
To start, these consolidated posts include a "Start-o-Meter" which is their attempt to consolidate all the advice. Basically, it adds together start, sit, start of the week, sleeper, bust, and sit of the week designations from a few different websites to come up with an overall start/sit advice. We can start there:
Scatter plot with regression line showing the correlation between the Start-O-Meter and the number of points scored by the player. We see a significant (p< 0.001) positive correlation between Start-o-Meter and Fantasy Points. Shading on the regression line represents a 95% confidence interval
As we can see there is a positive relationship, such that players with a higher Start-o-Meter rating score more fantasy points. But here is the thing, you could easily game this by saying start your studs and sit your duds. Also, positions are weird here, as a sit designation for a quarterback may still outscore a start designation for a kicker, while both were really correct.
We can therefore normalize a player's performance vs how they performed over the entire year. We can calculate the z-score by subtracting each player's season-long average performance and dividing by the standard deviation. This normalizes each players performance to itself, fixing both the start your studs gaming of the system, and the inherent differences in positional values. Intuitively, we would expect a start designation to indicate a better than average game from the player (a positive z-score) and sit advice to indicate a worse than average performance from the player (a negative z-score).
Scatter plot with regression line showing the correlation between the Start-o-Meter and the z-score points scored by the player. We see a significant (p< 0.001) positive correlation between Start-o-Meter and z-score Points. Shading on the regression line represents a 95% confidence interval.
Again, we see a nice positive correlation. A higher Start-o-Meter rating does indicate a better than average performance. Interestingly, the Sit advice doesn't really seem to be a worse than average performance. It should be noted that there is a lot of noise here, which despite the significant correlation gives me pause. The final aspect to analyze though would be how the start/sit players' actual performances compare to ESPN's projections. I'd hope a start designation would perform better than ESPN projected while a sit designation would perform worse (thereby providing value beyond ESPN's projections).
Scatter plot with regression line showing the correlation between the Start-o-Meter and the difference from ESPN projections scored by the player. We do not see a significant correlation (p=0.88) between Start-o-Meter and Difference from Projection. Shading on the regression line represents a 95% confidence interval
And here is the rub. We see a flat line when comparing the Start-o-Meter vs difference from projected. We would have liked to see start designations be above 0 while sit designations are below 0, but that just is not what we find. Again a very noisy dataset, but a Start or Sit designation does not seem to indicate a player is about to over- or under-perform their projection.
It is important to mention that Start-o-Meter is an amalgamation of a few different analysts (split between Fantasy Pros, CBS, Rotoworld, NFL.com, Bleacher Report, Sports illustrated, and ProFootballFocus). It could be that how we are combining the designations is the problem, so we can simplify it. If a player has a start-o-meter rating above 0 (which means more analysts have them as a start than a sit), then they get a start, if they are below zero, the get a sit.
Billing Department
Violin plots of how players perform compared to their projection when they have a start or sit designation overall. The inner thick lines represent a box plot with the white dot at the median value of the distribution. The vertical red line is the point where a player's performance matches their projection
Even here, we don't really see a difference from projected, if anything the start designation is worse (though this is guaranteed to be just noise, but gets the point across). As a reminder, this is not saying that players with a sit designation score more points than players with a start designation (we've seen above that this is not the case). But when compared to ESPN's projections, these start/sit designations do not seem to add any value. (Note we can check this statistically with a multivariate linear regression, regressing projected points and Start-o-Meter vs actual points. We find that the Start-o-Meter score accounts for less than 2% of the overall regressions predictive power)
One last aspect to look at would be the individual raters. Up until now we have been looking at how the consensus performs, but maybe some analysts are better at this than others? Maybe there is one analyst to rule them all? We can go and look at each of them individually. Below is violin plots of what the individual designations turn into vs espn's projections.
Violin plots of player performance compared to their projection vs their start/sit designation. Each graph is an individual analyst, with the white dot being the median and the redline being no difference from projections
Interestingly, we see some patterns start to emerge. The first is that for most of these they are actually pretty good at calling when to sit a player, we can consistently see the sit or bust designation predicting performances below a player's projections. In comparison, a designation of a start doesn't do much. Rotoworld's start of the week seems to predict a better performance than projected, but small sample sizes abound here. I'd also be a little wary of NFL.com's and CBS's articles as the different designations (outside of maybe the bust designation) don't seem to predict different levels of performance.
So finally, does one analyst rule them all? To determine this we can take the total difference from ESPN's projections of the start players and subtract the total difference from the sit players to create a total start/sit score. A high start/sit score would indicate that their starts outperformed their projections and their sits underperformed their projections, while a negative start/sit score would mean the opposite.
Boxplots of analysts start/sit scores. We see that no analyst really outperforms 0 (no difference from projected) but that Rotoworld seems to be a little better than the rest.
Looking across the analysts we don't really see a difference in performance across analysts. Yes, Rotoworld is a little higher, but not significantly so. Overall, the analysts seem to get as many start/sit designations right as wrong, and they have plenty of outliers in both directions.
Limitations
It is important to note the limitations of my analysis, in particular, small sample sizes here. while NFL and CBS mention a lot of players (1314 and 985 unique player/week combinations in 2019), other sites have fewer players (such as the 63 unique player/week combinations mentioned by Rotoworld). Considering the higher sample sizes are the ones that seem to have less predictive power may suggest that all designations will return to the mean, but only time will tell if that is true.
Putting it all together
In the end, I am not sure what predictive power the Start Sit columns add that your platforms player projections do not already provide. I have talked previously that ESPN's projections are relatively accurate and that certainly holds here. In the end, I think the value of the start sit articles is to provide you with the confidence to start or sit a player based on their projection, regardless of the name brand.
Questions? Comments? Let me know at ac@alexcates.com. Want to read more breakdowns like this? sign up for my newsletter. Finally, like what I do? Consider supporting me on buy me a coffee.