tanelpoder

Previous discussion about an earlier version of the same eBPF/bcc-based thread activity "tracking + sampling" concept and implementation:

- https://news.ycombinator.com/item?id=40869877 (2024)

The latest version uses modern eBPF/libbpf/Rust-based blazesym and in basic mode does not inject any tracepoints or probes onto the critical path of other threads in the systems (such passive sampling is achieved using sleepable eBPF programs, task iterators, etc).

Google's (Postgres-compatible) AlloyDB Omni also has similar functionality now - the main DB action, persistence, etc still has to land on persistent block storage, but additional data can be cached for reading on local NVMe disks.

Oracle's Exadata is a whole another beast (I co-authored a book about it back in 2010 and it has improved even further since then). It's a hybrid, shared storage distributed system - not consensus-based replication (although they support RAFT for global data systems now too), but a distributed, coherent buffer cache (global cache) based database system. As it's shared storage, you can write copies of blocks, WAL to multiple separate storage cells (NVMe or even just remote RAM) via direct RDMA operations, without OS kernel or system calls involved.

For analytic queries, yep Oracle can push down filtering, column projection, many aggregations and join filters (bloom filters) for early filtering into the storage cells. The bloom filters are used for early filtering of the next table in the join, based on the output of the previous query plan nodes so far.

Glad to be helpful! :-)

On Linux this is not true, on an IO heavy system - with lots of synchronous I/Os done concurrently by many threads - your load average may be well over the number of CPUs, without having a CPU shortage. Say, you have 16 CPUs, load avg is 20, but only 10 threads out of 20 are in Runnable (R) mode on average, and the other 10 are in Uninterruptible sleep (D) mode. You don't have a CPU shortage in this case.

Note that synchronous I/O completion checks for previously submitted asynchronous I/Os (both with libaio and io_uring) do not contribute to system load as they sleep in the interruptible sleep (S) mode.

That's why I tend to break down the system load (demand) by the sleep type, system call and wchan/kernel stack location when possible. I've written about the techniques and one extreme scenario ("system load in thousands, little CPU usage") here:

https://tanelpoder.com/posts/high-system-load-low-cpu-utiliz...

FWIW, I've used VMWare Fusion on Mac OS for years (and their Workstation product on Linux/Windows since 1999). It has worked well for me so far. VirtualBox used to be not great on MacOS with multiple CPUs and lots of interrupts - I noticed problems when moving from a Mac laptop with a spinning HDD to NVMe SSDs. The slow inter-processor interrupts (IPIs) and some related contention caused concurrent IOs to get very slow with fast SSDs with VirtualBox (back when still running on an Intel Mac with VMWare kernel module). In fact, I could get more IOPS out of the VM with fio when giving just 1-2 vCPUs to the VM, vs 4-8 vCPUs.

I haven't tested VirtuaBox on latest Mac versions as I'm happy with VMWare Fusion (also on ARM M1/M2 machines). The product is still good, regardless of Broadcom's business shenanigans.

Edit: If Parallels works for you, but only the SSH terminal sessions get stuck, maybe the "shared network" connection drops your connections for some reason. Before diving too deep into troubleshooting this across layers, you could try out the usual suspects workaround, setting SSH TCP keepalive. I have this for some connections in my ~/.ssh/config file:

TCPKeepalive yes

ServerAliveInterval 30

I found this article interesting (in fact, posted it earlier, but it didn't get traction then). I think some context is needed: When you operate at eBPF/kernel level you don't get easy direct access to the higher level goodies, like various container metadata (other than perhaps the cgroup id/name). So with eBPF you extract various numbers and IDs and then use userspace code+services to retrieve the meaningful (human-readable) context and strings using these IDs.

A plain Linux example would be that eBPF will only give you user/group IDs (uid/gid), not usernames, so you need to use post-processing in userspace code to convert these IDs into something meaningful.

After 8 years, worth reposting... Previous discussion here:

https://news.ycombinator.com/item?id=15474043

Yep, same here, first GUI editor that has ended up sticky enough. I've used just plain vim (or vi, back in the day) in a terminal. Occasionally tried out some GUI editors every few years, but none of them were suitable/sticky enough for my taste. Now I've been using Zed for 8-9 months, almost daily when coding vs. just testing/hacking something, and haven't fallen back to vim in a terminal for coding tasks yet.

The reluctance of using stored procedures where they’d be valuable is also a skill + will issue. I do get the non-database-developer view that if 98% of your app is gonna be written in some other language anyway, why complicate your app layers by adding a bit of PL/SQL code + testing infra here and there.

But for processing (and validating) various data loads and incoming data streams, by looking up matching values in existing reference tables, stored procedures can increase performance/efficiency and reduce complexity (of having to maintain connection pools and caching layers just for validating incoming record batches).

As far as the unwillingness to even evaluate stored procedures issue goes, I sometimes joke that maybe the DB vendors should start calling stored procedures as “database lambdas” instead…

Perhaps the grand vision was to later use all those newly built internal tools as reusable components in their customer-facing apps…

Yeah, I just wanted a cool-sounding name for this. Nevertheless, it allows you to do easy stress-testing with some vector search operations (a quite narrow set, but you can combine it with joins and write your own queries if you like). But "CatStress" didn't sound too good to me.

It's a "Vector Search Playground" really, but the bigger value so far has come from not running maximum stress tests, but demonstrating people how you can join vector search results to the rest of your (existing) application schema. Plenty of people have thought that you need a completely separate, isolated vectorstore behind some API for this...

Edit: Also the setup part includes running a "generate_embeddings.py" script that uses PyTorch under the hood (on CPUs or CUDA/GPUs) to generate embeddings from the 25k photos (or 9M when using the rotated variants). That process can also be sped up and optimized for sure - my whole point is that once everything runs OK enough from end to end, then it's time to start measuring and optimizing the whole process - for learning and fun.

I took a completely different path to visualizing embedding vectors’ physical layout [1] - mainly to explain how the data structures, data volumes and comparison would radically differ, compared to your regular btree index searches. I made sure to mention that you can’t make any conclusions based on just human eyeballing of these vector heatmaps, but the database people I’ve demoed this to, seem to have reached some a-ha moments about understanding how radically different a vector search is compared to the usual database index lookup work:

[1] https://tanelpoder.com/posts/comparing-vectors-of-the-same-r...

Yep, exactly - with OS level partitioning or logical volumes, you'd still end up with a single underlying block device (and a single queue) at the end of the day.

With SSDs that support namespaces you can use commands like "nvme create-ns" to create logical "partitioning" of the underlying device, so you'll end up with device names like this (also in my blog above):

/dev/nvme0n1 /dev/nvme0n2 /dev/nvme0n3 ...

Consumer disks support only a single namespace, as far as I've seen. Different namespaces give you extra flexibility, I think some even support different sector sizes for different namespaces).

So under the hood you'd still be using the same NAND storage, but the controller can now process incoming I/Os with awareness of which "logical device" they came from. So, even if your data volume has managed to submit a burst of 1000 in-flight I/O requests via its namespace, the controller can still pick some latest I/Os from other (redo volume) namespaces to be served as well (without having to serve the other burst of I/Os first).

So, you can create a high-priority queue by using multiple namespaces on the same device. It's like logical partitioning of the SSD device I/O handling capability, not physical partitioning of disk space like the OS "fdisk" level partitioning would be. The OS "fdisk" partitioning or LVM mapping is not related to NVMe namespaces at all.

Also, I'm not a NVMe SSD expert, but this is my understanding and my test results agree so far.

Yep indeed, that's a tradeoff, if your DB fits into a single volume. I'm not that deeply familiar with databases other than Oracle (that has its own ways to work work around this), so for ease of use, everything on a single volume keeps things simpler.

One thing that I try to achieve anyway, is to spread & "smoothen" the database checkpoint & fsync activity over time via database checkpointing parameters, so you won't have huge "IO storms" every 15 minutes, but just steady writing of dirty buffers going on all the time. So, even if all your files are stored on the same blockdevice, you'll less likely see a case where your WAL writes wait behind 50000 checkpoint write requests issued just before.

I'll see if I have a chance to run such a test on AWS in coming days (and would need to keep running it for much longer than just 5 seconds shown in the blog).

If you care about WAL write/commit latency, you could provision a small-ish EBS io2 Block Express device (with provisioned IOPS) just for your WAL files and the rest of your data can still reside on cheaper EBS storage. And you might not even need to hugely overprovision your WAL device IOPS (as databases can batch commit writes for multiple transactions).

But the main point is that once your WAL files are on a completely separate blockdevice from all the other datafile I/O, they won't suffer from various read & write IO bursts that can happen during regular database activity. On Oracle databases, I put controlfiles to these separate devices too, as they are on the critical path during redo log switches...

When doing 1M+ IOPS, you probably do not want to use OS IO schedulers due to the OS (timer & spinlock) overhead [1] and let the hardware take care of any scheduling in their device queues. But you're right about flattening the IO burst spikes via DB configuration, so that you'd have constant slow checkpointing going on, instead of a huge spike every 15 minutes...

All this depends on what kind of storage backend you're on, local consumer SSDs with just one NVMe namespace each, or local SSDs with multiple namespaces (with their own queues) or a full-blown enterprise storage backend where you have no idea what's really going in the backend :-)

[1]: https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-th...

Edit: Note that I wasn't proposing using an entire physical disk device (or multiple) for the low latency files, but just a part of it. Local enterprise-grade SSDs support multiple namespaces (with their own internal queues) so you can carve out just 1% of that for separate I/O processing. And with enterprise SAN arrays (or cloud elastic block store offerings) this works too, you don't know how many physical disks are involved in the backend anyway, but at your host OS level, you get a separate IO queue that is not gonna be full of thousands of checkpoint writes.

Throughput with enough I/O concurrency, yes. That's actually why I wrote this blog entry, just to bring attention to this - having nice IOPS numbers do not translate to nice individual I/O latency numbers. If an individual WAL write takes ~1.5 ms (instead of tens of microseconds), this means that your app transactions also take 1.5+ ms and not sub-millisecond. Not everyone cares about this (and often don't even need to care about this), but worth being aware of.

I tend to set up a small, but completely separate block device (usually on enterprise SAN storage or cloud block store) just for WAL/redo logs to have a different device with its own queue for that. So that when that big database checkpoint or fsync happens against datafiles, the thousands of concurrently submitted IO requests won't get in the way of WAL writes that still need to complete fast. I've done something similar in the past with separate filesystem journal devices too (for niche use cases...)

Edit: Another use case for this is that ZFS users can put the ZIL on low-latency devices, while keeping the main storage on lower cost devices.

I happen to be an Estonian, but this comment is general: I wonder when teaching kids to be AI-tool-literate, would this also help teaching some critical thinking and reading skills when consuming information from social media and Internet.

Since AI does hallucinate, you have to be able to dig deeper into the sources of facts when needed, to make sure that you're not being hallucinated lies. The same skills ought to be applicable for critical thinking/internet news literacy, as that loud "news" channel or TikTok influencer might also be "hallucinating" lies for you for their reasons...

I bought some Intel Optane persistent memory modules some years back to run some tests on them and maybe even build a PMEM-based KV store. I even wrote some articles about Oracle database's usage of PMEM for low-latency (local) commits [1]. But soon after I bought the memory, Intel discontinued the entire product line. I'm guessing that there just wasn't enough interest in practice as there were plenty of other (software-based) alternatives and NVMe SSDs got really fast too.

I've seen grand announcements, partnerships & alliances and unveilings of exciting CXL products that you can not order for the last 6 years by now. Or some CXL modules that add only 64 GB of CXL DRAM while taking 8x PCIe5 lanes. Perhaps there's lots of testing action going on with with select hyperscalers, but it's slightly worrying that after 6 years of announcements by major vendors (Samsung, SK Hynix, etc), you still can't just order a couple of these devices for your own testing.

I think shared, external CXL memory arrays that are connected to multiple servers (like SAN storage, but for cache-coherent, directly CPU-connected memory) would enable a few very interesting use cases for database engines:

1) Shared, cache-coherent global buffer caches for small-to-medium sized DB clusters

2) Shared build-side of giant hash joins for SQL (so you don't need to shuffle, repartition or broadcast data for your large joins)

Let's hope that this technology is real enough to become mainstream and actually available to users.

[1]: https://tanelpoder.com/posts/testing-oracles-use-of-optane-p...

Yup, one example that I noticed recently, RedHat backported the eBPF subsystem from Linux 6.8 to their 5.14 kernel (in RHEL 9.5):

https://docs.redhat.com/en/documentation/red_hat_enterprise_...

Blog entry: https://www.datadoghq.com/blog/ai/toto-boom-unleashed/

Yep, when you have thousands of different production apps, installed and running directly on Linux - not talking about containers or microservices here - you’ll have very little appetite to upgrade all of them to the latest and shiniest technologies every couple of years. Stability & compatibility with existing skillsets is more important.

Perhaps some companies do deeper analysis first - with AI summarizing & clustering signals from various internal systems and corporate email history. I'm not saying that this is better for everyone compared to current human-driven approaches, but the human driven approaches seem to be just about stack-ranking employees by their recent performance review ratings and drawing a red line in a spreadsheet. Amazon's recent "flattening the organization" initiative might well be using AI as one of the signals. I have no idea whether that's actually true - but then again it's 2025 and they have been a data-driven company for a long time.

Large corporations are full of middle-managers who do not lead anything nor produce anything (customer facing) and whose only job is to "facilitate information flow" and "pull teams together" etc. I think these roles will be replaced with AI chatbots quickly. The upside (for the company) ought to be that now the facilitation should happen very quickly and is measurable/manageable more directly. The downside (for the line employees) is that now your boss is a chatbot.

Also, anything you do with enterprise (cloud) customers. People like to talk about scale a lot and data people tend to think about individual (distributed) systems that can go webscale. A single system with many users is still a single system. In enterprise you have two additional types of scale:

1) scale of application variety (10k different apps with different needs and history)

2) scale of human capability (ingenuity), this scale starts from sub-zero and can go pretty high (but not guaranteed)

Astute observation, but also CrowdStrike would like a word :-)

Thanks. Yep I have to revamp the whole 0x.tools webpage, right now it's a mix of older tools & prototypes and the "final stuff" and it's confusing what's what.

The lsds verbose option shows where in the Linux /sys fs each individual field comes from (lsds -lpv) so that's the ultimate source of what each field means. But I could pull each sysfs file's description from docs into a table on the webpage (I'm probably too lazy to create a manpage for now - help is appreciated)

Edit: Since there are not that many fields, it would be possible to add a -d option in addition to -v to get a human readable description for each field too. One of the main sources of confusion is the "queue_depth" vs. "nr_requests" fields. My ideal (which I usually don't reach) is to make these tools "explainable", so that they tell you from where they got their input data (and what basic math was applied).

Hi, yep lsblk targets a wider area of functionality, like showing mountpoints, device UUIDs, while lsds focuses only on block device settings.

Maybe the latest Linux versions have lsblk versions that support these columns, but in RHEL9 at least I don't see equivalents to lsds'es WBT_LAT, QDEPTH (not the same as lsblk's RQ-SIZE), WCACHE, FUA and some others. But these 4 are which I regularly need (especially when troubleshooting a yet another slow fsync() issue etc). I did and do use lsblk all the time too, but still end up catting and grepping various additional files and correlating the results, sometimes on systems with 100+ multipath block devices.

The other reason was that I wanted a tool that shows me where it gets these values too (for myself and sometimes for explaining stuff to others).

Edit: That being said, it shouldn't be hard at all to add the said extra fields to lsblk too.