A/Bingo split testing now on App Engine, built for Khan Academy
bjk5.com
bjk5.com
1) The code is available under MIT and the docs are CCed and specifically designed to make it easy to clone this. At least three people have successfully done so, in a variety of programming languages, web frameworks, and technology stacks. If programming Java is not difficult for you, block off about a week or two and bam problem solved.
2) If you have access to vast amounts of cash but your programmers are just too busy delighting people to write an A/B testing framework, you could hire me to rewrite the whole codebase in Java. This would only cost you low five figures.
3) If you don't want to program and don't have tens of thousands lying around but do have tens lying around, Visual Website Optimizer is a great option for $CHEAP.
4) If you have neither programming skills nor tens of dollars then you may want to reevaluate whether A/B testing is a huge need at this point in your life.
http://www.fairtutor.com/fairlycertain/
I'd be surprised if you managed to take more than 5 minutes translating the core so that it compiled in Java. From there, it's just a matter of removing the ASP.NET usercontrols from the package and (optionally) wrapping it in your Java framework's equivalent.
http://www.evanmiller.org/how-not-to-run-an-ab-test.html 'If instead of deciding ahead of time, “this experiment will collect exactly 1,000 observations,” you say, “we’ll run it until we see a significant difference,” all the reported significance levels become meaningless. '
Reading the stats.py it looks like you might be doing this.
Thanks to Ben and Khan Academy!