Hacker News Reader

Comment by throwaway984393 | Hacker News Reader

I've work on projects where we did it the opposite way, with multi-tenancy as the default. It didn't work. For a couple customers, we had to carve out their own dedicated resources. A few of them were absolutely murdering the performance of the rest of the cluster(s), and some had business requirements to be completely isolated. Customers with similar requirements and workloads we kept in a multi-tenant pool.

Even though some of the benefit of multi-tenancy is (supposedly) simpler management (one set of resources), multi-tenancy can actually become more difficult as the customer pool gets bigger, or workloads get more uneven. Maintenance on the whole pool becomes more and more problematic, and you try to patch around or delay it by scaling vertically. You basically run into every possible problem and hit limits you hadn't thought of sooner than with single-tenancy. And worst of all, it's impacting more and more customers.

View on HN

Multitenancy means you can mix servers together, not that you must mix them.

Traffic shaping can help with customers that need a special SLA, Even routing some traffic to servers that no other users can access. You can also corral bad actors in the same way.

The secret is to do a hybrid thing. You’re completely multi-tenant but build it so there can be many copies of shared resources and associate a tenant with one specific copy of each.

So customer A is actually on load balancer #1, elasticsearch #3, db cluster #2, app server asg #4, etc.. Then when you need to carve out separate resources for VIP customers you just add a new number and only assign them to it.

Multi-tenant architecture with horizontal scaling!

That can work, but even that can get incrementally complex as you (or your customers' workloads) scale. There are also limitations between those components you mentioned that sometimes necessitate further separation. To deal with all that within the hybrid or multi-tenant architecture, you end up spending a lot more time on tooling just to be able to manage it, and that takes away time from more useful improvements and maintenance.

It's like spoon theory[1]. As a product team, you only have so many spoons. Every bit of difficulty you spend on maintaining a system's operation takes away spoons that could be used for other parts of the product. Regardless of whether you use multi-tenant or hybrid or single-tenant, if it starts to take away all your spoons, you should be ready to try a different model and see if you get some spoons back. (I think this applies to all aspects of a product team, not just operations)

[1] https://en.wikipedia.org/wiki/Spoon_theory

That is because you did not have 2 levels of multi-tenancy. first you have schemas inside one database. Next you can have multiple databases where these schemas live. If you want a single tenant to a database. Simply make a database with one schema for those resource hungry tenants. Tooling is key here. Java spring, for example, can do that. Not that easy though and requires a lot of knowhow for a lot of edge cases.

The main problem with multi-tenancy like that is migrations. It can take a lot of time. There are 2 strategies: backward and forward compatibilities. Or, green blue deployments with ISITIO or equivalent tooling (e.g. if one server is behind it is still routed to the old front-end).

> Not that easy though and requires a lot of knowhow for a lot of edge cases.

The key for implementing a multi-tenant applications in Spring is AbstractRoutingDataSource. It is not that hard, see https://stackoverflow.com/questions/49759672/multi-tenancy-m...

Still sounds like a multi tenant system just with some dedicated shards. This is fine as long as the code based and db schemas are in sync.

Reply on news.ycombinator.com