Awesome, I'm a huge fan of new and innovative tools that help improve the process of refactoring and improving existing code. This looks like a really promising tool for Ruby developers, and I'm always grateful when companies and their employees invest the time and effort to release their tools to the community. I really liked the point about "buggy data" as opposed to just buggy code, I think that's a really important point.

A few reactions from reading through the release:

Scientist appears to be largely limited to situations where the code has no "side effects". I think this is a pretty big caveat, and it would have been helpful in the introduction/summary to see this mentioned. Similarly, I think it would be nice to point out that Scientist is a Ruby-only framework :)

You don't mention "regression test" at any point in the article, which is the language I'm most familiar with for referring to this sort of testing. How does a scientist "experiment" compare to a regression test over that block of code?

Anyway, thanks again for writing this up, I'll be thinking more about the Experiment testing pattern for my own projects.