> * We used a mono repo and services only had to talk to other services from the same commit (or branch)
Huh, that was a new one, can you expand? How did that work in practice? Did all the services update to the latest commit automatically?
Huh, that was a new one, can you expand? How did that work in practice? Did all the services update to the latest commit automatically?
The deploy tool was home grown and was part of the repo. Having everything in a mono repo was really important because it dramatically reduced the number of permutations of interfaces that had to be tested down to one.
These trade-offs might not be right for everyone, but for us they were fine and allowed us to create a highly efficient development environment.
But what would happen if 1 of them failed to start? Or start, but immediately fail its tasks? Then you would have to revert all of them, at the same time, after some has already run for a short while, potentially writing data on the new format (for example). So I guess you can't avoid the problem with backwards compatibility completely..
Or, I guess, more realistically, you wouldl try to fix the failing service fast, without rolling back anything :)
So, if I knew the changes well, I could roll back if nothing in the API changed. We didn't like that, but sometimes you just need to be pragmatic. Other times an update was made that forced you to get it working. In this case services helped because rather than the whole system being down, only part of it was. So many of our services were background type work that in most cases, customers would never even know if they were down for even 1-2hrs.
We had good staging environments with a lot of reporting, monitoring and metrics. We never deployed to production code that had not run reliably in staging for a least a couple of days.