ecaml

However, you're right: in the article, it's not clear how we decided to consider the experiment as successful: we called a meeting with all the team and asked everyone something like: "Do you think that the situation on Slack is better now?", and the majority of people said "yes".

I added a note at the end of the article to address some of the comments ;)

You're right: scientifically/formally speaking this cannot be considered an experiment but, in my opinion, for small and easy-to-change processes, there is no need to be formal. The goal sometimes is just to solve small pain points and make people happier at work, and in such cases the fact that people perceive that things are better after the experiment could be enough :) Obviously, this was just a simple case study and every perceived improvement could be due to confounding factors, as you say, or just to the fact that the experiment created awareness about a problem. Obviously, if an experiment touches something more relevant than slack messages, being more formal is a good thing and A/B testing is for sure a better approach.

The point of the post was just to make people aware of the fact that changes in a team can happen without too much pain and that continuous improvement and experimentation are processes that can be implemented easily.