tanelpoder

I haven’t read the article yet, but back when I tried to get to over 100 GB/s IO rate from a bunch of SSDs on Zen4 (just fio direct IO workload without doing anything with the data), I ended up disabling Core Boost states (or maybe something else in BIOS too), to give more thermal allowance for the IO hub on the chip. As RAM load/store traffic goes through the IO hub too, maybe that’s it?

Author here, yeah indeed. Earlier I was worried if eBPF verifier would even let me do plain integer arithmetic and then just use it as a struct pointer to an "arbitrary" kernel memory location, but it works.

Another research/testing area is whether I need to worry about memory ordering and possibly add some memory barriers, especially on ARM platform, as the eBPF task iterator passive sampler is an outside observer, periodically running on a different CPU, not injected to the context (and critical path!) of the monitored threads via tracepoints/probes.

Related - about a year ago or so, I read about a firmware related problem with some vendors SSDs. It was triggered by some uptime counter reaching (overflowing?) some threshold and the SSD just bricked itself. It’s interesting because you could carefully spread out disks from the same batch across many different servers, but if you deployed & started up all these new servers around the same time, the buggy disks in them later all failed around the same time too, when their time was up…

One reason why I started blogging (back in 2007) was to have an easy reference for myself about some specific syntax or config parameter name that regularly came up when working as a consultant at different customers. And access my scripts I used for troubleshooting things. While I couldn't connect my own laptop to their networks to copy anything over, my public blog web-pages were accessible at almost every site went to.

One exception was when I visited a customer in Shanghai, it turned out that my purely technical blog was blocked in China (at least back then) as it was hosted on a wordpress.com subdomain. I then migrated it my own domain and it was accessible in China too.

The other reason is definitely tech-marketing for my business, but hopefully with 99% quality and 1% noise :-)

Yeah, ftrace is great and the trace-cmd frontend too. Another swiss-army-knife type of a tool, like perf... and is also available in older kernels.

One thing that I had missed was function call argument tracing, but looks like ftrace will soon have it too (update: it's actually available on modern kernels already, look for CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS kernel config option).

https://lwn.net/Articles/1003386/

https://lpc.events/event/11/contributions/1106/attachments/7...

I built this tool since I didn't just want to run single-table Postgres vector similarity search queries on some text snippets to demo vector search usefulness. I also didn't want to focus on some business vertical-specific example, so I downloaded 25000 pet images (Kaggle Cats and Dogs dataset) and vector-indexed their embeddings. And then built a simple web app that allows you to pick a cat or dog photo and find most similar-looking pet photos using Postgres HNSW vector search.

So that's just another way to demo what the higher-level "AI" similarity search means, but visually much more striking to people who are not that technical or are just getting started with this "AI" stuff.

Importantly, the web app also displays the SQL statements and execution plans (with performance metrics) that were executed in Postgres for generating the webpage. This makes some experimentation faster & easier as you could tweak the SQL or some parameter - create a different (vector) index - and just hit refresh in your browser to see the new results and the new execution plan with performance metrics.

My second goal was to blend together data from traditional tables (order history) and the newly generated embedding vectors, so I combined both worlds into a basic recommendation engine example. I first find top 20 most similar-looking cats compared to the selected one and then list the top purchased products by the 20 most similar-looking cats, all done within a single SQL query.

I finally wrote the "CatBench Vector Search Playground" formal launch post and posted it using "Show HN" here too:

https://news.ycombinator.com/item?id=42837246

Author here, thank you for the kind words :-)

I haven't yet written a "formal launch post" of the CatBench Postgres/pgvector app that I used for examples in the current article, but all the source code and install instructions are already available here:

https://github.com/tanelpoder/catbench

Yeah nothing wrong with “select distinct” itself if it’s used correctly for its intended reasons.

But when I see select distinct at the start of a big and complex query, I do immediately suspect that the developer might have missed some join condition down the line and “got too many rows” back from the query. And since the rows look like duplicates due to the missing join predicate, for a junior (or careless) developer, the quick “solution” is to just apply distinct in the top level query to get rid of these pesky duplicates, declare success and move on.

Context: With eBPF, you can have "wait events" even for database engines that do not have such instrumentation built in.

Showing examples of the "extended thread activity sampling" approach on DuckDB, MySQL, Postgres and Oracle.

My favorite (and also surprising) old Unix document is an USENIX paper from 1984, describing the /proc filesystem:

https://news.ycombinator.com/item?id=26298564

Unsinkable ship, with design flaws…

And “packet” loss!

Are there any good 3rd-party native frontend apps for Claude (on MacOS)? I mean something like ChatGPTs app, not an editor. I guess one option would be to just run Claude iPad app on MacOS.

An example of this tool in action (on Linux kernel source code) is here: https://sbexr.rabexc.org/

I once wrote a top-like tool in Oracle's sqlplus client, that is not designed for building self-refreshing terminal UI display apps. Just to see if I could do it, had to get creative too. Used pipelined PL/SQL functions with never-ending output stream and a sleep function within it and had to carefully match the sqlplus "fetch array size" with number of rows returned in a batch from the pipelined function. Called it MOATS - the Mother of All Tuning Scripts - and then someone took the idea further and built v2.0 with added colors and charts, etc:

The v2.0 UI GIF is here: https://github.com/dbsid/moats_rac

I recalled something from distant past - MSDOS multitasking with DESQView (apparently the same vendor who built QEMM):

https://en.m.wikipedia.org/wiki/DESQview

Does anyone remember NDISwrapper? You took Windows drivers for your laptop WiFi device or whatnot and the wrapper allowed to load them into Linux kernel (still didn’t work for me reliably though, not surprised). Even when I was young and had lots of free time, it seemed insane to have to deal with this.

Yep I should have written "mostly browser/SSH". But then occasionally I have to print something or run a (customer-provided) VPN client or open some corporate Excel file, etc. And there's the convenience factor of keeping my current setup. But should the MacOS thing become untenable for some reason, yep next step would be a Lenovo/System76 laptop known to work with Linux (including audio/bluetooth/wifi after resuming from sleep!) or even a Chromebook...

P.S. I've ran Linux/X (plus VMWare VMs with Windows) on my desktop machine with few complaints since the '90s and it was always the laptops that had issues, causing me to switch back to Windows after a couple of weeks of trial & frustration in the 2000s. I got Windows pretty performant & usable though, even attended Mark Russinovich'es Windows Internals class in London back in 2006 or so :-)

Yeah I sometimes use ripgrep for this. It’s suspiciously fast!

Well, you can run multitasking in TSRs & keyboard interrupt handlers (the original event loop)… implementing a window manager and TCP stack is left as an exercise to the reader…

Haha, off-by-one pixel error!

I still see MacOS as the best choice for my desktop/laptop uses (browser and SSH), but I also have a documents folder that I’ve accumulated over decades. I still use various .txt files in the docs folder as my low tech note taking apps.

I use the Spotlight or Alfred keyboard shortcuts (that also use spotlight index?) for quickly opening the files when needed - and annoyingly my most important file - notes.txt - regularly disappears from the Spotlight index and suggestions. It’s been like that for at least 5 years, probably closer to 10. I’m not even trying anymore, will just open the file from command line with vi as the fallback step.

Yep and ILOM access probably happens over the management network and can hardware-reset the machine, so the dataplane internode network issues and any OS level brownouts won't get in the way.

Yep, the "STONITH" technique [1]. But programmatically resetting one node over a network/RPC call might not work, if internode-network comms are down for that node, but it can still access shared storage via other networks... The Oracle's HA fencing doc mentions other methods too, like IPMI LAN fencing and SCSI persistent reservations [2].

[1] https://en.wikipedia.org/wiki/STONITH

[2] https://docs.oracle.com/en/operating-systems/oracle-linux/8/...

This reminds me of the Linux/Unix disk busy "%util" metric in tools like sar and iostat. People sometimes interpret the 100%util as a physical ceiling for the disk IO capacity, just like with CPUs ("we need more disks to get disk I/O utilization down!").

It is a correct metric when your block device has a single physical spinning disk that can only accept one request at a time (dispatch queue depth=1). But the moment you deal with SSDs (capable of highly concurrent NAND IO), SAN storage block devices striped over many physical disks or even a single spinning disk that can internally queue and reorder IOs for more efficient seeking, just hitting 100%util at the host block device level doesn't mean that you've hit some IOPS ceiling.

So, looks like the GPU "SM efficiency" analysis is somewhat like logging in to the storage array itself and checking how busy each physical disk (or at least each disk controller) inside that storage array is.

Cool, yep this is just a minor detail and doesn't change what the article itself conveys.

I'm guessing that at cloud scale, more innovation & scalability is needed for the control plane (not to mention the network itself).

Regarding a durable/asynchronously destaged write cache, I think EMC Symmetrix already had such a feature in the end of '80s or 1990 (can't find the source anymore).

The first diagram in that article is incorrect/quite outdated. Modern computers have most PCIe lanes going directly into the CPU (IO Hub or "Uncore" area of the processor), not via a separate PCH like in the old days. That's an important development for both I/O throughput and latency.

Otherwise, great article, illustrating that it's queues all the way down!

This could/should be a direction for companies like Zapier...

Edit: or Stripe.

I use SaaSPegasus [1] to avoid reinventing all the typical features, integrations and boilerplate code. It's based on Django. One interesting additional component that I recently discovered is PinesUI [2].

[1] https://www.saaspegasus.com

[2] https://devdojo.com/pines