Speedy transactions in multicore in-memory databases | Hacker News Reader

Speedy transactions in multicore in-memory databases

muratbuffalo.blogspot.com

110 pointszdw16 comments

So this design uses an epoch of 40 ms, does that mean any write operation will take on average 20ms?

Quite a lot of latency compared to regular databases.

I think so. Figure 7 is the only figure in the paper that shows latency, and it appears the epoch defines a latency floor. The system is aggressively optimized for throughput at the expense of commit latency, which is common for anything with group commit.

It supports multicore but it doesn't seem to support true distributed operation, which is what most applications require nowadays.

Interesting comment. Scaling "out" is costly in complexity and only needed... well... If it is needed!

700k transactions per second on a single machine from 2013 is probably millions per second on a modern machine.

We need to be careful and always ground the scale to reality, for example: the number of flight reservations per year worldwide is ~3B. Let's say you have 10 attempts / search per résa: 30B, then 3 transaction per, 90B. This is still less than 300 transactions per second in average!

I believe the lost art of squeezing out the performance of one machine is very valuable in today's world because a single machine can definitely achieve "most applications" at the human scale with no atomic clock to sync db transactions across the planet with thousands of nodes.

Scaling out isn't just about performance. It's also about high availability.

Which is something everyone needs to consider when using cloud infrastructure.

AWS provides 99.5% EC2 uptime guarentees which is ~2 days a year of outages.

That is not simply acceptable for most use cases and is why a single server just won't cut it.

> AWS provides 99.5% EC2 uptime guarentees which is ~2 days a year of outages.

> That is not simply acceptable for most use cases and is why a single server just won't cut it.

I agree with your point in principle, but I wonder why this should be applicable to most use cases.

Our national tax reporting system failed on the first day when you could submit taxes that year and has done so across multiple years - and yet everyone just submits their reports a bit later and it's fine in the end. Our e-health system users would be glad to have only a few days of outages per year, instead it doesn't work most of the time, due to the project being an abject failure that padded the wallets of many of who were involved in "making" it. Our national COVID vaccine signup system failed on the first day, even though they attempted to implement queuing of requests, most people were only able to sign up days later. You hear on HN about Cloudflare outages, Google outages, GitHub outages and so on occasionally.

I understand why you might want resilience for something like pacemakers or airplane systems, but surely most CRUD apps or cloud based services out there aren't actually that important, we just make it seem like they are, in fear of users going to competition during the outage or something.

The tax and eHealth systems have the advantage of being monopolies though, it's not like you can go to another tax service with better ICT (except by emigrating of course), and it's literally illegal to not use the system at all (because you have to pay your taxes), so you can't even choose not to use it. Most companies do have competitors, so "we are more reliable" becomes a selling point for them.

Definitely agree that some companies take their uptime demands beyond the point of diminishing returns though. I once worked at a company which would absolutely not allow 10 minutes of scheduled downtime in the middle of the night but didn't mind at all if I took several months to build a workaround.

Because engineers usually work for someone.

And that someone rarely accepts 2 days a year worth of outages given its impact on revenue.

Yeah, but flights are relatively expensive and aren't scheduled as often as e.g. social media messages are sent. Also, you always have to be prepared for bursts of requests. And of course some new requirements can always be thrown in by management which may invalidate any assumptions you made about speed.

Any data source where events are created by intentional human action, including global social media platforms, have relatively trivial event rates even during bursts. It isn't the event rate that gets you, it is the rest of the workload processing and serving content. Just landing and storing the events can be easily handled by a single modern server with capacity to spare.

Machine sourced data, by comparison, has event rates that are orders of magnitude higher.

I would love to see more of these kinds of discussions on HN. Where do folks go to be on the forefront data world (databases,algorithms, distributed systems, etc)?

If you're interested in academic papers, look at the top 2-3 conferences in each field (e.g. VLDB/SIGMOD for databases, SOSP/OSDI/USENIX ATC/EuroSys/NSDI/PODC(more theoretical) for systems/distributed systems. Some of the above might require an ACM digital library subscription to see the papers (which is really good value in my opinion), but others are completely free (i.e. the USENIX conferences). In many/most cases you can often find a copy of the paper online on the author's website.

all VLDB papers are also completely open access. You can find e.g. last year's papers here: http://vldb.org/pvldb/volumes/15.

> Where do folks go to be on the forefront data world (databases,algorithms, distributed systems, etc)?

Search for IRC channels on the topics of interest.

I'm currently going through one of the papers on https://www.nowpublishers.com/DBS and finding it good.

Some of these journals are behind a paywall. :(

If they're from publicly funded universities I'd wish the scholarly work would be free. I understand things coming out of Microsoft Research, for example, being pay-walled.

Reply on news.ycombinator.com