tanelpoder

Interesting. I just posted the link from this X post [1] without modifying it (and it wasn’t 0.0.0.0…). I think one thing HN does is that when it detects a HTTP redirect, it will start using the referred to URL instead…

[1] https://x.com/julianhyde/status/1982637782544900243

I submitted the link that Julian posted (to his article) on X yesterday. There’s indeed some refresh action in my browser too, but the article opens up.

(I submitted this link). My interest in this approach in general is about observability infra at scale - thinking about buffering detailed events, metrics and thread samples at the edge and later only extract things of interest, after early filtering at the edge. I’m a SQL & database nerd, thus this approach looks interesting.

One snippet: "At this stage, the remaining events are recorded to a 40 Petabyte disk buffer."

A 40 PB disk buffer :-)

Indeed, would be nice if there were a standardized API/naming for internal NVMe events, so you'd not have to look up the vendor-specific RAW counters and their offsets. Somewhat like the libpfm/PerfMon2 library for standardized naming for common CPU counters/events across architectures.

The `nvme id-ctrl -H` (human readable) option does parse and explain some configuration settings and hardware capabilities in a more standardized human readable fashion, but availability of internal activity counters, events vary greatly across vendors, products, firmware versions (and even your currently installed nvme & smartctl software package versions).

Regarding eBPF (for OS level view), the `biolatency` tool supports -F option to additionally break down I/Os by the IORQ flags. I have added the iorq_flags to my eBPF `xcapture` tool as well, so I can break down IOs (and latencies) by submitter PID, user, program, etc and see IORQ flags like "WRITE|SYNC|FUA" that help to understand why some write operations are slower than others (especially on commodity SSDs without power-loss-protected write cache).

An example output of viewing IORQ flags in general is below:

https://tanelpoder.com/posts/xcapture-xtop-beta/#disk-io-wai...

Good article, thanks for sharing. I've been working on one part of this problem space for quite a while too. I want ability to directly drill down into latency reasons and underlying application component threads' wall-clock time, instead of having to correlate various systemwide utilization metrics and try to manually connect the dots.

I'm using eBPF-based dimensional data analysis, starting from bottom up (every system is a bunch of threads, including distributed systems) and move up from there. This doesn't replace existing distributed tracing approaches for end to end request view, but gives you deep observability all the way down to each service's underlying threads' wall-clock time (where blocked, sleeping and why, etc).

At this year's P99CONF I will launch the first GA release of my (open source) 0x.tools xcapture eBPF collectors, with a reference implementation of a TUI tool, showing dimensional performance modeling on these new thread sampling signals (xtop).

A couple of 1-minute asciicasts of xtop are here: https://tanelpoder.com/posts/xcapture-xtop-beta/

I guess it's because SimpleFold came from a research lab with different autonomy and less competing interests and internal politics...

Yeah looks like this will be a common theme :-)

This wasn't on Linux, but on one of the old-school commercial Unixes - a customer had memory leaks in some of their daemon processes. They couldn't fix them for some reason.

So they invested in additional swap space, let the processes slowly grow, swap out leaked stuff and restart them all over the weekend...

Link to paper: https://dl.acm.org/doi/pdf/10.1145/3748355.3748367

Interesting, works for me (3 different browsers on the same mac).

Oracle's blogs use some (JS based?) dynamic retrieval of the content, it's not all in "index.html" apparently, perhaps you have an ad blocker or something which prevents the dynamic load.

Indeed. Previously, when computing latency histograms from sampled events, I've abused DuckDB SQL cross join/cartesian products to generate zero values for "empty buckets" within a single SQL statement [1]. But makes the SQL unnecessarily complex & slower too, so I quickly moved that empty bucket rendering functionality to the frontend...

[1]: Example DuckDB SQL with generated bucket lists and cross join + outer join: https://github.com/tanelpoder/0xtools/blob/master/tools/sql/...

Asciinema is one of the best products/tools I’ve ever used. I mean the auth/upload CLI flow is so straight to the point and easy to use, despite having to cross a few layers (yep I know other CLIs have something similar, it’s just that Asciinema has never “gotten in my way” when doing something with it)

I think if you click "edit" after submitting and manually edit the word to caps, it will be accepted then.

A previous discussion from 2024 is here (39 comments):

https://news.ycombinator.com/item?id=40869877

I have since converted the entire codebase to use modern eBPF functionality (task iterators, task-local storage maps, sleepable iterators capable of reading user-space memory of other processes, etc), with libbpf-based userspace frontend.

Optane memory modules also present themselves as separate (memory only) NUMA nodes. They’ve given me a chance to play with Linux tiered memory, without having to emulate the hardware for a VM

If this link doesn't work, just go to https://google.com/ai - it redirects to https://www.google.com/search?udm=50&aep=11 (HN strips the udm= field apparently).

This URL allows you to jump straight to the AI chat mode, without having to search for something first.

Yep agreed, for single-host with CXL scenarios. I wrote this comment thinking about a hypothetical future CXL3.x+ scenario with multi-host fabric coherence where one could in theory put locks and control structures that protect shared access to CXL memory pools into the same shared CXL memory (so, no need for coordination over regular network at least).

Yup, for best results you wouldn't just dump your existing pointer-chasing and linked-list data structures to CXL (like the Optane's transparent mode was, whatever it was called).

But CXL-backed memory can use your CPU caches as usual and the PCIe 5.0 lane throughput is still good, assuming that the CXL controller/DRAM side doesn't become a bottleneck. So you could design your engines and data structures to account for these tradeoffs. Like fetching/scanning columnar data structures, prefetching to hide latency etc. You probably don't want to have global shared locks and frequent atomic operations on CXL-backed shared memory (once that becomes possible in theory with CXL3.0).

Edit: I'll plug my own article here - if you've wondered whether there were actual large-scale commercial products that used Intel's Optane as intended then Oracle database took good advantage of it (both the Exadata and plain database engines). One use was to have low latency durable (local) commits on Optane:

https://tanelpoder.com/posts/testing-oracles-use-of-optane-p...

VMware supports it as well, but using it as a simpler layer for tiered memory.

Yeah I saw the same. I've been keeping an eye on the CXL world for ~5 years and so far it's 99% announcements, unveilings and great predictions. But the only CXL cards a consumer/small business can buy are some experimental-ish 64GB/128GB cards that you can actually buy today. Haven't seen any of my larger clients use it either. Both Intel Optane and DSSD storage efforts got discontinued after years of fanfare, from technical point of view, I hope that the same doesn't happen to CXL.

... and if you have the money, you can use 3 out of 4 PCIe5 slots for CXL expansion. So that could be 2TB DRAM + 1.5TB DRAM-over-CXL, all cache coherent thanks to CXL.mem.

I guess there are some use cases for this for local users, but I think the biggest wins could come from the CXL shared memory arrays in smaller clusters. So you could, for example, cache the entire build-side of a big hash join in the shared CXL memory and let all other nodes performing the join see the single shared dataset. Or build a "coherent global buffer cache" using CPU+PCI+CXL hardware, like Oracle Real Application Clusters has been doing with software+NICs for the last 30 years.

Edit: One example of the CXL shared memory pool devices is Samsung CMM-B. Still just an announcement, haven't seen it in the wild. So, CXL arrays might become something like the SAN arrays in the future - with direct loading to CPU cache (with cache coherence) and being byte-addressable.

https://semiconductor.samsung.com/news-events/tech-blog/cxl-...

Yes, can confirm, the book is great. I was also happy to see that the author correctly (in my mind) used the term “embedding vectors” vs. “vector embeddings” that most others seem to use… Some more context about my pet peeve: https://tanelpoder.com/posts/embedding-vectors-vs-vector-emb...

When I'm on the fence about some (technical) decision, I use a "razor": if all options seem equal, go with whichever is the simpler one. The results are ok so far and it has been great for reducing my brain-energy spent on pontification and early optimization too far ahead.

I liked the post, but these kinds of articles do make sense to people who've already been through the trenches & view the advice from their seasoned experience PoV and apply it accordingly. But if people without such experience follow it to the letter just because it's written, can have surprises ahead.

I haven't tested Intel's efficiency cores (E-cores) myself - would these address the need for desktops/laptops?

Also, latest Oracle version (23ai) has added "concurrent on-commit fast refresh" functionality, where concurrent transactions' changes are rolled up to the MV concurrently (previously these refreshes were serialized).

https://oracle-base.com/articles/23/materialized-view-concur...

From the article: "Concurrent refreshes will only happen if there are no overlaps in the materialized view rows updated by each session."

In the database-nerd world, we had something like this about ~10 years ago, written by @flashdba. Still a good read:

https://flashdba.com/category/storage-for-dbas/understanding...

I wrote the CatBench vector search playground toy app exactly for this reason! [1] ("cat-similarity search for recommendation engines and cat-fraud detection"). I built it both for learning & fun, but also it's useful for demoing vector search functionality, plugged in to regular RDBMS application schemas in business context. I used cats & dogs as it's something everyone understands, instead of diving deep into some narrow industry vertical specific use case.

[1]: https://tanelpoder.com/posts/catbench-vector-search-query-th...

Pretty cool. At some point I wrote my own routines with pure DuckDB SQL, so I wouldn't have to import UDFs, it would all be in just SQL files. DuckDB already has a basic `bar()` function for bar charts [1] and also you could use YouPlot [2] if you prefer just piping the output to some other command instead of using a library or extension. I created my own bar charts though, as I wanted to use the "partial" unicode block characters for more fine-grained bar charts.

I created my own sparklines too with LISTAGG and unicode characters for rendering latency histograms, but the bigger problem there was that some latency buckets didn't have any data falling into them. That missing row of data still needs to be rendered as an empty bucket. So while the pure SQL approach worked, it was a convoluted query where I used an outer join with a list of synthetically generated latency buckets to align the rest of the buckets right.

That was a prototype anyway, later on I switched to returning the filled bucket numbers (latency:count) with LISTAGG as CSV that I later parsed and rendered in the app layer [3]

[1] https://duckdbsnippets.com/snippets/54/quick-inline-bar-char...

[2] https://duckdb.org/docs/stable/guides/data_viewers/youplot.h...

[3] https://tanelpoder.com/posts/xcapture-v3-alpha-ebpf-performa...

Nice flashback to the '90s... Back in 1999, I worked with a 3 TB sized Oracle DB that was supposedly the biggest Oracle database in Europe on Windows! 32-bit Windows NT 4.0 running on a machine with 16 GB of RAM, so we had to use Oracle/Windows very large memory hacks to "swap" physical pages in to the 32-bit address space as needed...

Edit: Yes, we had blue screens every couple of weeks

I have used Internet Explorer 4 on Solaris 2.6 for SPARC. I don’t remember why…