348 karma · joined March 3, 2017
1) Being able to clone the author's blog repo and read it offline (just a bonus that it's already in org mode so no post-processing necessary). Is there a "lemme download everything by this author and read offline?" button on Medium?
2) The blog being in a public Git repository is nice for transparency.
It was crafted by a student, Kenneth John Freeman, for his Cambridge dissertation published in 1907. Freeman did not claim that the passage under analysis was a direct quotation of anyone; instead, he was presenting his own summary of the complaints directed against young people in ancient times. The words he used were later slightly altered to yield the modern version."
Thanks for sharing this the-dude, you've helped make me a little less misinformed, even though I'm sad to learn it's not a direct quote :)
I'd suggest thinking about the following BEFORE YOU RUN A SINGLE A/B TEST:
1) Key Metrics: Define these. They are the general, "I don't care what your experiment is about, these numbers are important." Every experiment you run should automatically track these metrics. You should also give the ability to define custom metrics, since an experiment that changes some random button color probably wants to look at how many people clicked the button, which is almost definitely NOT a key metric.
2) Logging Infrastructure: Make sure that you have a easy-to-use, reliable data pipeline set up for logging and processing events. Bad logging == bad experiment results. Also consider streaming vs batch processing for updating experiment results.
3) Population Management: How do your experiments segment users? Are variants calculated in realtime? Batched with some SLA for lag? Are they sticky?
4) Mutual Exclusion: People running experiments often want "their" users excluded from other experiments.
5) Guardrails: Do your experiments automatically shut off if there is a catastrophic decline in one or more key metrics? What safety measures do you have around determining if an experiment is safe/valid? How do you handle cleaning up data when there's a problem? What sorts of actions invalidate an experiement's existing results? Does your entire site break if your A/B Testing service is down for whatever reason?
6) Cleanup/Ownership: Experiments don't run forever (at least they shouldn't!). Cleaning up old features, populations, etc can a pain, especially when the people that wrote the stuff originally no longer work at the company. Make cleanup mandatory and as easy as possible.
There's a lot more, but I'm tired now. A/B testing is complex. There are lots of resources out there, though. Look for white papers on the subject, they're surprisingly approachable. Example from Microsoft: https://exp-platform.com/Documents/2017-08%20KDDMetricInterp...
To not be purely depressing, I'll present an example of something I think is a step in the right direction: At least some universities are good about giving out scholarships to low-income students that offset the difference between in/out-of-state tuition.