Fun with NFL Stats, Bokeh, and Pandas

Thanks for the link! That's a great resource and write up about the statistical heaping effect. I'll include that in a post update.

ha! that's funny. "We've got to get to the 35 yard line, so Morten can kick a 53-yarder, so we can have a chance."

There are multiple other factors involved that aren't discussed fully in that article.

To start the ref bias is not necessarily unintentional. Measuring first downs and exact placement of the line of scrimmage becomes more difficult and takes more time when the ball is not initially spotted on an 5 yard increment or individual hash mark. The league offices might directly instruct refs to spot the ball on an exact yard line if there is any doubt of the spot in order to speed up the game. An obvious example of this intentional bias is when the ball is punted out of bounds. It is nearly impossible for a ref to get an exact spot in that situation and yet it is almost always spotted exactly on a hash mark.

Also lots of drives will start on a set yard line and penalties are often handed out in 5 yard increments. So while there are no rules that will start a drive on the 15, there are rules that will start a drive on the 20 or 25 and a standard 5 or 10 yard penalty would place the ball exactly at the 15 yard line.

Lastly, the spot is a continuous data point that is being recorded by humans as a discrete data point. That means the herding could be exaggerated in the recorded data and not necessarily as pronounced during the actual game. A ball might be spotted by the ref at 14.4 yards but the person responsible for data entry might eye the spot and record it as being at the 15 yard line.

It also could be bias in the person recording the data, who may unconsciously prefer rounding placements between the 4 and 6 yard lines to 5.

While I have no explanation for the clustering, a yard is a long, long measure in football, with a dozen 4K cameras and a lot of very vested interests. Occasionally spots seem generous or not, but that's to the measure of a couple of inches (e.g. he went down 4" short of the first down). As a long time fan it just doesn't seem reasonable to me that the ref opts for the 40 yard line instead of the 39 because there's a line. That's a 3 foot difference. To further support this, the bordering two yard counts do not depress the amount of the excess on the 5 yard marker.

Not to mention that where the refs place the ball -- near the center of the field, usually on the left or right side depending upon where the play ended -- has a line on every yard marker.

But again I have no explanation. We know that touchbacks start on the 20 or 25 yard line, and penalties are increments of 5 yards (e.g. face mask would be 15 yards, holding 10 yards, false start 5 yards, encroachment 5 yards, etc...and these can pile, kickoff out of the last five yards starts at the 40 yard line, etc), and to some degree this explains the 5 yard excess.

I've often wondered if there was a way to detect bias in officiating. Naturally there's going to be subjective element to the types of penalties called, but I also have wondered if there was a way to detect bias in the situations in which penalties are called and how they affect the probability of victory.

Additionally, with the whole Tim Donaghy [0] NBA thing, I have always been curious if there was a way to detect referee influence on the outcomes of games based on the line. There certainly times when Vegas loses (a few weeks ago was particularly bad for them), but it would be interesting to see if there was a way to detect probabilities of officials, coaches, or even players deciding the outcome of matches beyond the course of the endeavor.

But really, it'd be fun to have statistical backing to confirm or refute the whole "College Football has a pro-East Coast bias" or "this referee always gives fouls to the European soccer sides", or "the Patriots _always_ get those calls because ... Tom Brady."

[0] https://en.wikipedia.org/wiki/2007_NBA_betting_scandal

Agreed. And I definitely make no claims about this being earth-shattering "data science". I just happened spend a few hours over the weekend making some plots and commenting about what I saw with some Python tools.

I'll also state that I am neither a data scientist nor a statistician. I'm a Python application engineer with a background in mechanical engineering, so that might help set the context a bit more.

In my experience, certainly a large number of jobs advertised at 'data scientist' would be exactly as you describe. Emphasis on the 'cleaning' part.

Data Analyst

Thanks. And I agree with Bokeh being overkill for this. My original intention was to make it fully interactive but I hit some snags on keeping the interactivity through Pelican SSG so I just kept 'em static.

That project is no longer maintained because I don't use it any more, but others have picked up the baton: https://github.com/derek-adair/nflgame

Back in the day, I used nflgame along with

    https://github.com/BurntSushi/nfldb
    https://github.com/BurntSushi/nflvid
    https://github.com/BurntSushi/nflfan

to setup a simple local web UI that allowed me to quickly search through every play and watch any single play I wanted. Video footage was available as soon as the game was over, and play info was available live as the game was playing. It was amazing.

This worked because nflvid downloaded full HD NFL games from their CDN, which was unprotected at the time. (I paid for an NFL Game Pass subscription and never distributed the video footage.) They also had XML files that delineated the time at which each play started and its duration. Some ffmpeg slicing and dicing was all it took to cut up a full game and associate each clip with each play. That's all part of what nflvid does.

I hacked all of this together in my free time years ago, and I bet a lot of people would find it amazing. One wonders why the NFL doesn't build this and sell it themselves. When I used Game Pass a few years ago, you could search for plays with rudimentary criteria, but only over a single game at a time. It was artificially very limited.

> Just in case author sees this, this is wrong, right? It should read Running becomes more and more popular as you use up your downs?

Er, no, the author is right; the share of all plays that are passing plays goes up with down number (until dropping at 4), the graph shows that quite clearly.

Something like this?

http://www.yardsgained.com/#(passes_~_sacks)_~_first_down_at...