896 karma · joined August 17, 2018
I am not claiming I have the answer or the French laws are wrong, just presenting a data point. Nevertheless, this is HN, so I expect people how dumb I am and how I have no idea what I am talking about. Fire away.
So, if I knew the changes well, I could roll back if nothing in the API changed. We didn't like that, but sometimes you just need to be pragmatic. Other times an update was made that forced you to get it working. In this case services helped because rather than the whole system being down, only part of it was. So many of our services were background type work that in most cases, customers would never even know if they were down for even 1-2hrs.
We had good staging environments with a lot of reporting, monitoring and metrics. We never deployed to production code that had not run reliably in staging for a least a couple of days.
Most of the services just ran normally on a developers system, or we had a docker script that could spin things up locally, or you used one of the development sandboxes running on the system.
BTW, we had a system that allowed very low cost creation and management of containers (VMs in our case). I think that is an important detail.
Could you use a debugger for a feature? Most likely not, because a lot of our features already pulled data from external sources which you had no visibility into.
> You're talking like there was a single "owner" for each service? What did you do when they went on holiday / left? How did people share knowledge about the system?
Same problem every development team has. Who setup up the CI server? Who setup the front-end proxy? Who configured the AWS setup? Everything needs redundancy. We had a list of all tools and services and had multiple names assigned to each one. Even without services a team should have this.
> Dealing with an outside service brings a lot of overhead. It's harder to debug, harder to test. When things are maintained by a separate team it makes sense to treat them as an outside service, but within the same team why would you make all that extra work for yourself?
Because it worked really well and we didn't see that extra work. I disagree that external services are hard to interface to. They are hard to work with when they are buggy, but if they hit the specs, they generally are not an issue. I have no problem interfacing to S3. Because of the loose coupling, things were very extensible and easy to modify. A few times we would replace a service with a new one by adding the new one and slowly migrating everything over to use it. In some cases, that migration took over a year, but that was fine.
> Sounds like you need a better language. I agree that anything that might segfault belongs in a separate service, but most of the time the solution is to not use stuff that might segfault.
LOL, it was Python. Bugs happen, things break. Everything can eventually segfault for all kinds of reasons. Run a complex system for a decade and you will see all types of issues. I remember running OpenVZ before switching to KVM and occasionally we would even see kernel panics.
> How does that fit with "services only had to talk to other services from the same commit"?
You conveniently left out "or branch". Listen, I am not trying to sell anything, I don't care what you think. I am just sharing my teams experience. It was overwhelmingly positive using services to build a complex system with real revenue from Fortune 100 customers for almost a decade.
The deploy tool was home grown and was part of the repo. Having everything in a mono repo was really important because it dramatically reduced the number of permutations of interfaces that had to be tested down to one.
These trade-offs might not be right for everyone, but for us they were fine and allowed us to create a highly efficient development environment.
* We used a mono repo and services only had to talk to other services from the same commit (or branch). We didn't version service APIs, I think that would have been a huge waste of time for us.
* Our tests did unit tests and integration tests.
* We could use different languages as the problems required. We started one data processing service in Python and migrated it to Go when it became more CPU bound than we were comfortable with. We have an old but extremely reliable C++ service that had been running for almost a decade.
* Each service ran in its own container. When a container/vm would run out of memory, we all didn't have to scramble to figure out who broke something. The person who owned the code for that service dealt with it. If a service started kicking out errors nobody but the owner had to be distracted.
* We had a good team of architects that understood how to build things. We didn't split services across atomic boundaries. We knew how to build interfaces that handled errors robustly. If you don't know how to do these things, you service APIs can quickly become a mess.
* The world is services. Just think about AWS and how it works. There is no such thing as a monolith, eventually you interface to services. Be it a weather API, a CI server, deployment system, a central logger, etc. Just because you didn't write all those services doesn't mean you don't have a service architecture.
* We had really disjoint services that would have made zero sense to put together. For example, the web server that handled the front-end vs a back-end data system that refreshed a product database and built PDF assets based on the data in overnight batch runs. We actually had many of the later for different customers. We had services for central logging, for monitoring, for metric storage, etc. No way would I ever want to push all that into a single service and debug a GC issue or segfault.
* If we had to hotfix a service, we could deploy just that service with zero concern the other services would have an issue.
For us, services were a key part of building a very reliable system that we could update rapidly with minimal risk. We did this for almost a decade. We had a very good group of engineers and never did I hear one of them say services were holding us back. After this experience, I would say everyone one of them is an advocate for the appropriate use services.
As a customer, I tried buying compute sticks in large quantities (i.e. 1000s) from a distributor but they would not extend pricing discounts until Intel approved them. In order to get approved, I had to take online training classes and quizes Intel offered. It was insane. No other company I dealt with required anything like that. I spent hours doing online training on their solutions and never was able to arrange a single discount. Finally said screw it and went with a different solution. Hands down the worst company I dealt with.
The guy who coded Alta Vista gave a talk at Stanford that I was able to sit in on. The most interesting story was when a person contacted him and begged him to remove something from the index because his boss was going to see it the next morning. He wrote a filter just to take care of this guy. It was fun stuff back then.
For my skill level and effort put forth, I don't know what I could have used to would have worked as well.