2,885 karma · joined March 14, 2012
Co-founder, Kami - kamiapp.com
This is a difference from PG, where having all the writes be concentrated in the key range is helpful as the Btree nodes covering that will be more likely to be highly cached
We did have to make some schema changes in order to improve the data locality and avoid hotspots - CockroachDB did _work_ without doing those, but the changes improved performance massively. It wasn't super time consuming though, and the admin UI/monitoring in cockroach is nice for showing you any hot ranges.
The biggest one was our table which stores annotations on a document - on PG it's primary key was only (annotation_id UUID), and we had a secondary index on (document_id). This meant that on Cockroach the data for a document was spread across many nodes and querying all the annotations for a document would take ~40ms. We changed the primary key to (document_id, annotation_id) which co-located the data and that came down to ~2ms. Composite primary keys aren't ideal for ActiveRecord but the performance win was worth it.
We also had some tables with hotspots which was mostly just a matter of making sure they used uuid keys. It would be a problem though if we had a table with individual rows which were super hot or where the table didn't have enough data to get split into multiple ranges - though cockroach does have load based sharding to help here. It was only really necessary to worry about the higher load tables here, if it's below ~2000 qps or so it won't matter.
I looked at a bunch of options but ultimately managing a sharded database seemed like too much overhead for our small (but expanding!) dev team - so we decided to move our heavily loaded tables to CockroachDB instead, as since it's mostly Postgres compatible our transition would be easier, and it would be much easier to manage as it automatically balances load and heals. We're running cockroach on GKE and it's a really nice fit for running on kubernetes.
Ended up working well for us - we still have our smaller tables on PG but we want to move it all to Cockroach over time. There were some teething issues as we learnt what to monitor on it (read amplification is the big one here, though pebble has helped a lot vs the previous rocksdb), and it does need a lot of hardware at high scale (we're running 50 16-vcpu nodes for it) but overall I'm happy with it.
There's been so many queries where I've thought 'that's going to need a join and aggregation across tens of billions of rows, no way!' - and then Clickhouse spits back a query result in 10 seconds...
The issues recently with localstorage being broken ( https://twitter.com/jaffathecake/status/1389493762129375232?... ) have been particularly egregious, there's lots of users losing content on safari and we have to explain it's due to Apple breaking their browser (again).
We did switch to using binary queries though (by passing result_format as 1), and at that point we did see a speedup from using bytea columns - but it’s quite a bit of work to get the type mapping correct then so probably not worth it in most cases
That said over time as the hosted k8s solutions have gotten better Heroku has comparatively become less compelling, plus while Heroku can get to a large scale, you will eventually start to run into limitations with what you can run inside a dyno which would force you to move to something else. Since you'll have a 12-factor app though, it's relatively easy to migrate, the hard part is finding a new platform which doesn't feel like a massive downgrade vs Heroku.
We used Heroku for years and were spending around $2k per month (having increased from $1K a few months before) when we migrated the last stuff to GKE and our spending dropped to nothing instantly. I was pretty surprised to have nobody at all reach out to us to see why our spending had changed... Now we spend 7 figures on GCP instead.
Heroku solved a lot of problems incredibly well but they seemed to lose any ambition to be more than a salesforce deployment platform a few years after the acquisition and as k8s and other solutions have gotten better they just don't have a super compelling proposition now. I still use them to spin up side projects though as they still provide the best completely out-of-the-box experience.
We're back in NZ and focusing solely on the education market which really accelerated our growth. Not sure when we'll be able to travel to the US again but would be great to catch up when that happens!
If you need automated deployments, centralized logging, autoscaling, etc which many teams do, then you're going to be dealing with a bunch of complexity anyway.
Honestly baffled at how many people are convinced you need hundreds of engineers before k8s can work well - they must be doing something very different from what we are. We did keep databases out of k8s (until recently when we added CockroachDB running inside the cluster) and only had ~6 services to move which may have kept things simpler. We're now scaling to run thousands of vcpu at peak times and a dozen different services, and it's still not all that hard to manage.
Can somebody who has had the opposite experience comment on what actually made it so difficult to implement? I imagine that if you are managing the cluster control plane yourself that will make things much more difficult - but unless you have some very specific requirement you can use a hosted k8s to reduce that.