Hacker News Reader

Comment by Aurornis | Hacker News Reader

> Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoue

They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?

The article then ends before saying if the new setup solved their problems.

It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.

I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.

View on HN

> The article then ends before saying if the new setup solved their problems.

They got to put ClickHouse, DuckDB and Trino on their resume & LinkedIn. Problem solved!

A while back, a client called me in to look at a system whose author has left years ago. It was a mess. And for no good reason, it used Docker extensively, and would only work with a specific pre-release version. I searched and discovered that the guy had stayed just long enough to do a talk on this then-hot new technology before getting a better job elsewhere.

It was the clearest example of Resume Driven Development I'd ever seen, but no means the only one.

I worked at a startup that locked all the architects cronies in a room for a number of months to build every white paper he was interested in. Told rest of company to adopt this new world order… within a year of the rewrite they laid off 2/3 of engineering and sold the company off shortly thereafter. Architect accepted a role at FAANG for experience gathered. I remember a lunch with him where he joked “I usually leave companies worse off than when they started.” Self-indulgent psychopath.

Screw that company mate. I was in a company with startup-y mindset. It was just like that and I quit after 3 days, without taking any money

Honestly that shouldn’t even matter anymore.

The things the original commenter wants to know are exactly the things that will count in an interview. The reasons behind the decisions. Not the actual implementation.

Implementing anything, even something of significant complexity is trivial with frontier agents now. So build your experience / knowledge in a technology-agnostic way. The language/app/stack is superficial.

Edit: Speaking anecdotally; it’s what I’d look for in a candidate if I was hiring today

Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.

Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?

The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.

If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.

> Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.

Counter-point - the longer you delay a crucial decision the more information you have to make it a correct one, so you might avoid migrating DB engine twice a year

I suspect you and the person you are responding to are actually in agreement, making that an additional "point", rather than "counter-point".

The crucial choice is not (just) the one made in a pivot: choosing any DB (or any similarly foundational stratum of your system) is always a crucial decision. Ultimately, you have two choices: put off making an informed, optimal choice, or make an informed, optimal choice now - or if not now, then as soon as you have enough data/experience to make the optimal choice for your needs.

I think you might be attributing the bulk of potential cost to the act of migrating away from an initial choice. But the cost of migrating away from something is proportional to the extent to which it is intertwined/coupled with your system, which in turn is largely a function of the scale at which you are operating. The problem, at least as I have seen it, is that organizations often pick something with minimal (if any) thought put into that initial choice, wait until scale reaches a point that switching away is matter of life and death, and then have to desperately fight against inertia. I think that's what the parent of your post is getting at.

Once the avalanche starts it is too late for the pebbles to vote.

Yes, when decision can be delayed, that is powerful. But we are discussing decisions that have already been made which need to be unmade or made again. Where it breaks is when the tech debt is growing faster than our expertise. The usual stall is not to allow time to apply resources to the problem but to continue to not apply resources to the problem. So our knowledge base is growing at a negligible rate. Particularly if you’re a first mover. Nobody will solve your problems better than you have until it’s far too late to migrate to their solutions.

Founder here -- this is about our warehouse product which is separate from our main analytics DB. Our DuckDB warehouse was used by a handful of pilot customers. Our clickhouse database is serving all of our analytical queries.

If this is a pilot program for a small number of users, the blog post would feel a lot more logical.

As it was written, it looks like the engineering team is migrating the core infrastructure of the company around.

Try VictoriaLogs for your warehouse product next time. It provides efficient storing and querying petabytes of wide events with hundreds of fields - see https://docs.victoriametrics.com/victorialogs/keyconcepts/ , and it provides very easy scaling path from single-node setup to cluster setup, so you can start with an instance per customer and then gradually scale to many storage nodes as workload grows - https://docs.victoriametrics.com/victorialogs/cluster/#singl... .

It reminds of people who use ORMs, never pop the hood to analyze the queries, indexes, etc - then decide it's the database that's the problem.

There are many good reasons why the industry largely moved away from having DBAs as a specialist position. But one downside I've seen is that it's increasingly common to have teams decide to just switch platforms without fully exploring whether that's really necessary.

I'm not really surprised. Data storage and data access technology is a pretty large subject all by itself, and a "jack of all, master of none" developer might never develop the expertise needed to properly optimize a database.

with the caveat being that I came into the industry in 2014 so I may have missed the golden age of the DBA fully - I worked for a DBMS vendor on sales/solutions side for a while and most DBA’s I spoke to were slightly more knowledgeable business analysts. I met very few who really understood IO and Paging and Locks and how they’d impact a DB’s performance. Which kind of makes sense because if you were really obsessively into that you’d probably do something different or you’d work for a business dealing in storage systems. The value a DBA has been very low in my career outside of forcing process improvement just by saying no to things.

Yeah. I’d say that by 2014 a lot of the shift had already happened.

>The article then ends before saying if the new setup solved their problems.

This reads to me like a recruiting post; thats why they are open if it solved their problems. I'm not a fan of the way they framed it, it seems they have made their architecture decision and now need help implementing this.

I'd imagine if you are the kind of person that wants to work on large scale data query problems this is an invite for you to apply. That said, if you are that person it might not be ideal to join after all the fun decisions have been made.

What's wrong with ClickHouse again?

I don’t know? They have blog posts about it but I don’t know how much I’d trust them because their chosen replacement only lasted 5 months before the next replacement.

Russian ties?

https://news.ycombinator.com/item?id=33082224

That's where it started very long ago, but they are based out of the Netherlands mostly now.

https://clickhouse.com/blog/we-stand-with-ukraine

> We have no infrastructure, servers or people in Russia. The original engineering team successfully moved out of Russia and are now working in our Amsterdam office.

You can take the guy out of Russia.. :)

I kid but that's probably what people have a problem with.

Could say that about almost any country in the world, especially USA :)

Sure can. But some countries are more different like China, Russia, North Korea. A lot of people would include USA in that list of course. :)

Unless something has changed in the last year, a lot of teams have serious problems with stability.

I am curious to learn more about those issues, anyone have shared their experience publicly? would love to read

PostHog has listed a lot of job openings in the last couple of years. I figured growth but maybe it’s turnover.

I think it was closer to what they need; it comes down to keeping TB of data on local SSD and having a query engine next to it. That takes persistent, shared CPU, caching, sharding, resource management, etc.

Duckdb is an interesting approach to resource management (all ephemeral), but TB's are TB's. I doubt it was finding data where it needed it.

Clickhouse seems slow at getting to the top end of scaling; it's stuck with the 99% of its userbase doing 1 GB queries. I'm hoping they put some of their money into it.

ClickHouse is fast OLAP. Just do Iceberg with Trino like a normal big data person. WTF?

DuckDB is now close to competitive with Trino. As the post shares, large distributed multi-tenant data systems require a ton of work no matter what technology you use, but as a huge DuckDB fan, I do have to agree that it is not an out of the box data warehouse, though check back in a year or so when DuckLake and DuckDB 2.0 have had time to mature a bit more for this use case.

As a huge duckdb fan, I can’t think of a workload set I have where the two overlap, so I don’t see them as competitive - I would either pick Trino/Presto or DuckDB as the obvious solution.

No support for real S3 (zero copy replication) in the open source version.

Sounds like a common problem in the AI era.

It doesn’t look like planning and then executing.

It looks like executing and simultaneously reacting to the periodic “novel” information that would have been surfaced during the planning.

Burn enough tokens and it gets harder to tell them apart. Sometimes. But it’s appearance more than anything else.

Failure to plan can’t be repaired with even more tokens.

Reply on news.ycombinator.com