HNHacker News
TopNewBestAskShowJobs

jordanthoms

2,885 karma · joined March 14, 2012

jordan@thoms.nz

Co-founder, Kami - kamiapp.com

submissionscomments
jordanthoms··on Update on Omicron
You can strike New Zealand from that list, our isolation systems for people arriving were never very good and didn’t stand a chance once delta became dominant.
jordanthoms··on Scalable PostgreSQL Connection Pooler
We’ve been using this for a while after we found pgbouncer became a bottleneck - it’s so much faster and has been totally reliable. Thanks!
jordanthoms··on Memory leaks are crippling my M1 MacBook Pro
Ok... so what's the laptop that can compete with the new 14 and 16 Macbook Pros then and doesn't have some stupid flaw? The issue with waking up in bags seems to be caused by Modern Standby which is on most new Windows laptops?
jordanthoms··on Dune 2 is officially happening
2001: A Space Odyssey is (imo) the greatest film of all time, and it's just about incomprehensible on first viewing and explains nothing. So while it might not be the best commercial move I'm personally very happy with telling the story with less dialogue.
jordanthoms··on MacBook Pro 14-inch and MacBook Pro 16-inch
Not my experience at all, you must have something weird in your setup. For a full dev environment I'm running containers with postgres, cockroachdb, several ruby apps, kafka, some JVM based stuff, and clickhouse and it does hit battery life a bit but certainly not pegging the CPU, actually it's using less than Slack...
jordanthoms··on Firefox Now Sends Your Address Bar Keystrokes to Mozilla
In retrospect it seems firefox phone was actually a good idea, it’s been reborn as KaiOS and doing great. Pretty embarrassing for Mozilla to fail and cancel the project and then have another company take it and succeed…
jordanthoms··on Lessons learned from sharding Postgres at Notion
I think that might be counterproductive on cockroach - the ideal key has locality for data accessed together, but also has writes generally spread across the key range. If all the new rows have a similar key prefix then the shard covering the end of the key range will be hot which limits scalability.

This is a difference from PG, where having all the writes be concentrated in the key range is helpful as the Btree nodes covering that will be more likely to be highly cached

jordanthoms··on Lessons learned from sharding Postgres at Notion
The transaction isolation level hasn't been an issue for us - Our read/write patterns aren't super 'transactional' which may be why, we don't get a lot of conflicting updates as usually only one user has the permission to edit their content on the document. Arguably these tables could be on Scylla/Cassandra with eventual consistency even, though that would have been a harder transition and we'd have to spend more time making sure our application could handle that correctly.

We did have to make some schema changes in order to improve the data locality and avoid hotspots - CockroachDB did _work_ without doing those, but the changes improved performance massively. It wasn't super time consuming though, and the admin UI/monitoring in cockroach is nice for showing you any hot ranges.

The biggest one was our table which stores annotations on a document - on PG it's primary key was only (annotation_id UUID), and we had a secondary index on (document_id). This meant that on Cockroach the data for a document was spread across many nodes and querying all the annotations for a document would take ~40ms. We changed the primary key to (document_id, annotation_id) which co-located the data and that came down to ~2ms. Composite primary keys aren't ideal for ActiveRecord but the performance win was worth it.

We also had some tables with hotspots which was mostly just a matter of making sure they used uuid keys. It would be a problem though if we had a table with individual rows which were super hot or where the table didn't have enough data to get split into multiple ranges - though cockroach does have load based sharding to help here. It was only really necessary to worry about the higher load tables here, if it's below ~2000 qps or so it won't matter.

jordanthoms··on Lessons learned from sharding Postgres at Notion
At Kami (kamiapp.com) we had to make a similar transition last year - we'd seen usage grow over 20x in a 6-month period on our classroom product due to the pandemic, and we were storing all our content in a single-primary Postgres database. We had plenty of headroom before but we were suddenly running out of write capacity, even after scaling vertically up to ~150 vcpu. Our workload is super write heavy so scaling out replicas wouldn't have helped much at all, and I was spending a lot of time manually vacuuming in our daily off-peak as we were burning transaction ids at an astonishing rate, ~800 million per day.

I looked at a bunch of options but ultimately managing a sharded database seemed like too much overhead for our small (but expanding!) dev team - so we decided to move our heavily loaded tables to CockroachDB instead, as since it's mostly Postgres compatible our transition would be easier, and it would be much easier to manage as it automatically balances load and heals. We're running cockroach on GKE and it's a really nice fit for running on kubernetes.

Ended up working well for us - we still have our smaller tables on PG but we want to move it all to Cockroach over time. There were some teething issues as we learnt what to monitor on it (read amplification is the big one here, though pebble has helped a lot vs the previous rocksdb), and it does need a lot of hardware at high scale (we're running 50 16-vcpu nodes for it) but overall I'm happy with it.

jordanthoms··on ClickHouse, Inc.
TEXT/JSONB are stored inline IIRC unless they hit a certain size limit at which point they'll be put into TOAST - so you'll see data in clickhouse but some big values might be missing.
jordanthoms··on ClickHouse, Inc.
We recently setup Clickhouse on GKE using the Altinity operator (and signed up for Altinity support).

There's been so many queries where I've thought 'that's going to need a join and aggregation across tens of billions of rows, no way!' - and then Clickhouse spits back a query result in 10 seconds...

jordanthoms··on I can't stand developing for Safari anymore
It’s a mix of chrome and firefox, though also some safari to make sure we notice issues. What I’m talking about is obviously broken web apis ( another recent example: https://news.ycombinator.com/item?id=27509206 ), or missing features, not so much compatibility issues that would be explained by that. We have far fewer issues from Firefox for example, and certainly not major data loss bugs like those.
jordanthoms··on I can't stand developing for Safari anymore
Yeah, this. ~95% of our usage is on chrome (our app is education so lots of chromebooks) and yet safari makes up 50% of the problems.

The issues recently with localstorage being broken ( https://twitter.com/jaffathecake/status/1389493762129375232?... ) have been particularly egregious, there's lots of users losing content on safari and we have to explain it's due to Apple breaking their browser (again).

jordanthoms··on Rhizome – A pedagogical example of a JIT for Ruby, implemented in Ruby
JS used to be thought of as a slow language, then better VMs came along and now it's fast. I think it's much more a function of how much resources have been expended on developing a fast VM than the fundamentals of the language, though it will always be much harder to make a fast VM for Ruby than something like Lua.
jordanthoms··on An early look at Postgres 14: Performance and monitoring Improvements
Yeah re the binary base64 encoding - we found that when using text based queries (the default for pg gem on ruby and most other client libraries AFAIK), base64 in a text column outperformed byte columns as it avoids extra conversions to and from base64.

We did switch to using binary queries though (by passing result_format as 1), and at that point we did see a speedup from using bytea columns - but it’s quite a bit of work to get the type mapping correct then so probably not worth it in most cases

jordanthoms··on The Decline of Heroku
That's OK for a side project, but Heroku's selling point is that it's both easier to setup than that, and also has the resiliency, scaling and features you need for a proper production app. What happens if that EC2 instance dies suddenly or your EBS disk disappears, do you have point-in-time recovery for that DB? What about if you get an influx of traffic and suddenly need to scale beyond what a single server can do (which is admittedly pretty high). How about logging, deploys of a live copy of your site for each feature branch, etc? You can of course solve all those problems, but to solve all of them and do it as well as Heroku does will be a lot of work.

That said over time as the hosted k8s solutions have gotten better Heroku has comparatively become less compelling, plus while Heroku can get to a large scale, you will eventually start to run into limitations with what you can run inside a dyno which would force you to move to something else. Since you'll have a 12-factor app though, it's relatively easy to migrate, the hard part is finding a new platform which doesn't feel like a massive downgrade vs Heroku.

jordanthoms··on The Decline of Heroku
I think the real mistake there is that they waited until that high bar was hit before reaching out - if they'd started the conversation earlier they might have figured out what discounts they'd have to offer to keep the business, got them on a long term contract etc.

We used Heroku for years and were spending around $2k per month (having increased from $1K a few months before) when we migrated the last stuff to GKE and our spending dropped to nothing instantly. I was pretty surprised to have nobody at all reach out to us to see why our spending had changed... Now we spend 7 figures on GCP instead.

Heroku solved a lot of problems incredibly well but they seemed to lose any ambition to be more than a salesforce deployment platform a few years after the acquisition and as k8s and other solutions have gotten better they just don't have a super compelling proposition now. I still use them to spin up side projects though as they still provide the best completely out-of-the-box experience.

jordanthoms··on The Rise, Fall and Revival of AMD (2020)
USB4 is essentially a more open Thunderbolt, so I think their strategy will be to implement that instead.
jordanthoms··on RIP Flynn.io
Thanks, we are doing well - never did get into YC but it was a valuable process every time we applied - gave us time when we're not occupied with the day-to-day operating to think about our strategy.

We're back in NZ and focusing solely on the education market which really accelerated our growth. Not sure when we'll be able to travel to the US again but would be great to catch up when that happens!

jordanthoms··on RIP Flynn.io
Yeah, that's probably a major factor here - we moved to google cloud specifically for GKE and it's been pretty great. Have had a couple of issues (mostly with kube-dns autoscaling not keeping up, node-local dns helped a lot), but ultimately it's saved far more work than it's created.
jordanthoms··on RIP Flynn.io
We don't have review apps yet, we just have a few testing environments which devs can push changes they want to have available to. It is a really nice feature of heroku though so something it'd be nice to replicate. I think it'd probably take a week or so of dev time to do, the hardest part would be automating spinning up copies of all the dependencies we need.
jordanthoms··on RIP Flynn.io
Still wear my flynn shirt from 2015 when you kindly did a YC practice interview with us! Thanks for that BTW, y'all captured the intensity of it very well :)
jordanthoms··on RIP Flynn.io
How do you suggest companies which don't need binpacking run their workloads, and is it really that much simpler than using k8s?

If you need automated deployments, centralized logging, autoscaling, etc which many teams do, then you're going to be dealing with a bunch of complexity anyway.

jordanthoms··on RIP Flynn.io
Sure, but the advice people generally give is that small teams and startups shouldn't use k8s - and those are the organizations who can most likely use one of the hosted k8s implementations (which are available anywhere AWS/GCP/Azure have datacenters). Larger corporates will have more complexity to work in with but also will have larger teams which can handle that.
jordanthoms··on RIP Flynn.io
I moved us from Heroku to GKE a few years ago when we had ~4 engineers - Took a bit to get things figured out and get CI deploys etc working well, but it really wasn't all that complicated.

Honestly baffled at how many people are convinced you need hundreds of engineers before k8s can work well - they must be doing something very different from what we are. We did keep databases out of k8s (until recently when we added CockroachDB running inside the cluster) and only had ~6 services to move which may have kept things simpler. We're now scaling to run thousands of vcpu at peak times and a dozen different services, and it's still not all that hard to manage.

Can somebody who has had the opposite experience comment on what actually made it so difficult to implement? I imagine that if you are managing the cluster control plane yourself that will make things much more difficult - but unless you have some very specific requirement you can use a hosted k8s to reduce that.

jordanthoms··on Daft Punk Break Up
Hah, Get Lucky is the only song I skip on the album, overplayed for me.
jordanthoms··on GameStop Is Rage Against the Financial Machine
He already cashed out $14M today, so that's a 280x locked in.
jordanthoms··on Thou shalt not run a database inside a container
Yeah, joining the chorus but this is outdated info. We're running a CockroachDB cluster on top of GKE and the management aspect is excellent. For something like Postgres it's more difficult and we still run that outside of containers, but I'm sure it can be done well also.
jordanthoms··on Build your own professional-grade audio amp on the sort-of cheap (2018)
There's plenty of extremely high quality class D amps around now that exceed the performance of Class A, e.g. the Benchmark AHB2 ( https://www.audiosciencereview.com/forum/index.php?threads/r... )
jordanthoms··on Bleeding-edge tech will kill your startup
Good at smaller scale, and Heroku can go pretty good at scale. But eventually you run into things that they make harder rather than easier if you need to do something which doesn't fit well with their model. Deploying onto Google Kubernetes Engine has a bit of boilerplate setup work, but after that it's not a huge management overhead considering everything it gives you.
← PreviousPage 2 of 27Next →