For A/B tests implemented in backend code, I just use rand() to bucket a user, then store that bucket in redis.

After it's been live for a while, I pull the data from redis then use this open-source calculator to view the results: https://thumbtack.github.io/abba/demo/abba.html

For anything more complicated (e.g. split test shared across multiple systems, A/B testing multi-step funnels) I fall back to Kissmetrics.

With a ton of assumptions, mainly that you have an existing data or analytics pipeline, a simple in-house experimentation framework could take on the order of weeks rather than months to build.

There are no open source alternatives for the wysiwyg editor part of optimizely. But before you roll your own backend implementation have a look at https://facebook.github.io/planout/ .

(We had success in combining planout with a cms for the visual part. Marketing could do simple a/b tests. More complicated tests where done in the backend together with a developer.)

Not a full alternative, but long ago I created this super-simple A/B testing tool you can use on top of Google Analytics - not back-end required!

https://github.com/danmaz74/ABalytics

I already posted this on another thread[0]. Please check it out and happy to answer any questions or accept PRs.

[0] https://news.ycombinator.com/item?id=16357600