jessicaraygun

Yes! This is very important to us in dealing with transactional email.

Our Z-Test Method in Confidence.js pinpoints when results become statistically significant by calculating the required sample size at any given time using the standard error, zScore, and margin of error. If we have more samples than required, we can stop our test with confidence that we have statistically significant results.

We're still working out a way to do this with the new method - we have a hunch it might have to do with identifying when the confidence level becomes stable (as it does with more samples over time).

Thanks for your question! I'll pass it on to Emily, our stats expert.

Hey! I'm the author of Confidence.js. Emily Malcolm and I have been working hard on this new approach for the past few weeks and we're super excited to share it!

We're both here to answer any questions :)

Hey martian, thanks for the feedback. Your methodology is pretty sound - it looks like we use a similar technique. If you supply a small sample size here, you'll get the message, "There is not enough data to determine a conclusive result".

The recommendation of 95% significance level is quite common across A/B testing resources. The main goal here is to be able to say which variant will consistently perform best (in this case, 95% of the time).

I agree that it would be useful to be able to change the confidence level based on a site's specific needs - this is a feature I am looking to implement in Confidence.js soon.

Hey Noel,

I was in charge of the math here - I really appreciate your feedback. You seem really knowledgeable in this area and I would love to improve this A/B testing library.

If you have some time to chat, email me at jessica@sendwithus.com

This is a feature that I would love to add to Confidence.js sometime in the near future.

Hey, I'm the author of Confidence.js which was used to power this A/B testing calculator. That sounds really interesting and worth looking into - thanks for the feedback!

You're right about revenue being a very important metric, and I think optimizing for conversions has its place too. I would be interested to hear the results of your repurposing!

Glad you like it, happy to have you contribute :)

Stats class R sleepy... Real world applications like A/B testing are definitely more interesting.

We did a lot of digging and there aren't a lot of resources that are easy to digest - but this one really stood out to us:

http://visualwebsiteoptimizer.com/split-testing-blog/what-yo...

I'd love to write a summary of everything I've learned - maybe a blog post in the near future!

Hi sharnett,

Thanks for bringing that to my attention - I agree that our explanation is a little more convoluted than it needs to be. I'll update that later today to be more "readable" ;)

Thanks for the feedback!

Confidence.js comes from how we operate on A/B tests at sendwithus - we think of them as variants associated with results. While the code could definitely boil down to a few simple math functions, we find objects to be a good way to organize our data models.

Hey! I'm the author and I'd be happy to answer any questions you might have.

Confidence.js is based on the A/B testing code that we use at sendwithus.

A/B testing math is hard and we've worked really hard on making this great. Let us know what you think!

The layout generator is so cool! Yoink.