I'd suggest thinking about the following BEFORE YOU RUN A SINGLE A/B TEST:
1) Key Metrics: Define these. They are the general, "I don't care what your experiment is about, these numbers are important." Every experiment you run should automatically track these metrics. You should also give the ability to define custom metrics, since an experiment that changes some random button color probably wants to look at how many people clicked the button, which is almost definitely NOT a key metric.
2) Logging Infrastructure: Make sure that you have a easy-to-use, reliable data pipeline set up for logging and processing events. Bad logging == bad experiment results. Also consider streaming vs batch processing for updating experiment results.
3) Population Management: How do your experiments segment users? Are variants calculated in realtime? Batched with some SLA for lag? Are they sticky?
4) Mutual Exclusion: People running experiments often want "their" users excluded from other experiments.
5) Guardrails: Do your experiments automatically shut off if there is a catastrophic decline in one or more key metrics? What safety measures do you have around determining if an experiment is safe/valid? How do you handle cleaning up data when there's a problem? What sorts of actions invalidate an experiement's existing results? Does your entire site break if your A/B Testing service is down for whatever reason?
6) Cleanup/Ownership: Experiments don't run forever (at least they shouldn't!). Cleaning up old features, populations, etc can a pain, especially when the people that wrote the stuff originally no longer work at the company. Make cleanup mandatory and as easy as possible.
There's a lot more, but I'm tired now. A/B testing is complex. There are lots of resources out there, though. Look for white papers on the subject, they're surprisingly approachable. Example from Microsoft: https://exp-platform.com/Documents/2017-08%20KDDMetricInterp...