HNHacker News
TopNewBestAskShowJobs

brendangregg

4,953 karma · joined September 27, 2011

http://www.brendangregg.com
submissionscomments
brendangregg··on The return of the frame pointers
If I call the same "get statistics" command over and over in a loop (with zero queries), or 100% the same invalid query (to test the error path performance), I believe we'd call that a micro-benchmark, despite involving a full database. It's a completely unrealistic artificial workload to test a particular type of operation.

The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.

brendangregg··on The return of the frame pointers
Thanks!
brendangregg··on The return of the frame pointers
From the docs: "pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over"

While it might call itself a benchmark, it behaves very microbenchmark-y.

The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.

brendangregg··on The return of the frame pointers
Ah right, thanks, I remember I saw Andrii's analysis in the other thread. https://pagure.io/fesco/issue/2817#comment-826636
brendangregg··on The return of the frame pointers
Those are microbenchmarks.
brendangregg··on The return of the frame pointers
Thanks; what was the Python fix?
brendangregg··on Ubuntu 24.04 LTS will enable frame pointers by default
I don't know where 1-2% comes from, but for many scale production workloads I studied it was so close to 0% that it was tough to measure beyond noise on the cloud. That's not to say that 1-2% is wrong, but that it's likely someone's workload and other people see less.

Helping people find ~30-3000% perf wins, helping debugging and automated bug reports, is huge. For some sites it may be like 300 steps forward, one step back. But it's also not the end of the road here. Might we go back to frame pointer ommision one day by default if some other emerging stack walkers work well in the future for all use cases? It's a lot of ifs and many years away, and assumes a lot of engineering work continues to be invested for recoving a gain that's usually less than 1%, but anythings possible.

There's a couple of problems with an apt reinstall. One is that people often don't work on performance until the system is melting down -- many times I've been handed an issue where apt is dreadfully slow due to the system's performance issue and just installing a single package can take several minutes -- imagine reinstalling everything, it could turn the outage into over an hour! The other is I'd worry that reinstalling everything introduces so many changes (updating library versions) that the problem could change and you'd have no idea which package update changed it. If there was such an apt reinstall command, I know of large sites (with experience with frame pointer overheads) that would run it and then build their BaseAMI so that it was the default. Which is what Ubuntu is doing anyway.

brendangregg··on I bought a CO2 monitor and it broke me
I've had a few CO2 meters, and learned not to trust any <$100 as they don't really work. I look for meters that use non-dispersive infrared diffusion sensors (NDIRs), like the Aranet4. I've found the TIM10 desktop model from co2meter.com to be accurate (AFAICT), uses NDIRs, and only US$139.

I also have other air quality meters. (I collect measuring devices.) I wish there was a do-it-all air meter.

brendangregg··on iOS Ships Dvorak, Finally
Why not dvorak one-handed (both right and left)? I've tried it when I've injured one hand badly, and it's optimized for one-handed typing.
brendangregg··on Is bin-opening in cockatoos leading to an innovation arms race with humans?
Random cockatoo/FAANG story: I had an all-hands meeting earlier this year at Netflix, with me remote in Sydney, and a cockatoo knocked on my glass office door wanting to be let in -- so I'm saying "stop it! stop it!" when I realized my mic was on, and I'd said that to the whole org as my new manager was speaking. I quickly explained to everyone that I wasn't criticizing the meeting, oh no, I was actually talking to a bird that was knocking on the door.

I then felt it best to post a video of the bird knocking so people didn't think I was crazy. They knock with their beak: tap tap tap. Gets annoying when they do it at 6am to wake you up.

Recently I have noticed cockatoos raiding the bins in Sydney, it's definitely a thing.

brendangregg··on Shouting in the Datacenter (2008) [video]
FWIW I did a "making of" video soon after, in 2009: https://www.youtube.com/watch?v=lMPozJFC8g0 . Only bit I missed discussing there was that I did try to download freeware frequency generators wondering if there was some "rust note" of HDD harmonics, but those I downloaded didn't work and out of frustration I decided to start yelling instead.
brendangregg··on Brendan at Intel.com
Yes, a Sun Microsystems training room in Sydney. The entire building is now gone. I was teaching sysadmin and performance classes there in the early 2000s, for both internal and external staff.
brendangregg··on Brendan at Intel.com
Thanks, I forgot where that one was.
brendangregg··on Brendan at Intel.com
Thanks! We have to work through each capability carefully. Some won't be safe, and will be available on bare-metal instances only. That may be ok, as it fits with the following evolution of an application (this is something I did for some recent talks):

  1. FaaS
  2. Containers
  3. Lightweight VMs (e.g., Firecracker)
  4. Bare-metal instances
As (and if) an application grows, it migrates to platforms with greater performance and observability.

The ship has sailed on neighbor detection BTW. There's so many ways to know you're a VM with neighbors that disabling PMCs for that reason alone doesn't make sense.

brendangregg··on Brendan at Intel.com
Yes, getting PMCs enabled in VMs was just the start, I think the next hardware capabilities to enable are:

  - PEBS (Precise/Processor event based sampling, so that we can accurately get instruction pointers on PMC events)
  - uncore PMCs (in a safe manner)
  - LBR (last branch record, to aid stack walking)
  - BTS (branch trace store, " ")
  - Processor trace (for cycle traces)
Processor trace may be the final boss. We've got through level 1, PMCs, now onto PEBS and beyond.
brendangregg··on Magic-trace – High-resolution traces of what a process is doing
Thanks, good to see a new perf tool (and not just another procfs top clone :-). I've started using processor trace in the cloud, thanks to bare-metal instances, but for years I couldn't touch it (not available in VMs). There's a wealth of new information it provides, and we need better tooling on top of it, like magic-trace.

Glad to see "overhead" mentioned and quantified. I'd put the 2-10% at the top though, as that's heavy handed for some environments (can trigger a production fail-over).

I see magic-trace has implemented what some call "flame charts" (time on the x-axis) and not "flame graphs" (alphabet on the x-axis). The best tools do both (e.g., TraceCompass). Please do both! Will make seeing the big picture easy (flame graphs) and then zooming into time-based patterns easy too (flame charts).

brendangregg··on Netflix End of Series 1
Ah right, people have asked about that before. I'm just happy with a single normal-sized monitor.

Maybe related, but we once went around the Netflix parking lot with Blippar to look at the KBB value of the cars, and found that I was driving the cheapest car at Netflix (at least until we got sick of looking). Mine was a 2001 Toyota Prius, KBB value ~$1800. Not the cheapest car to ever park at Netflix, just the cheapest on the day!

brendangregg··on Netflix End of Series 1
I understand why people might think that, but I'm not sure it'd be great for the culture for me to be different from my coworkers. We're all working on important things, and we all need to collaborate together.
brendangregg··on Netflix End of Series 1
Yes, I helped organize the Netflix cricket practice sessions. Netflix has a cricket team (in engineering) that in the past has played in the GPL: Google Premiership League, along with several other tech companies. Lots of fun!

We practiced on baseball diamonds, and played league games on the Google soccer field.

When I ran the BPF tutorial at USENIX LISA 2019, I had one finger in a splint from breaking it playing cricket. :-)

brendangregg··on Netflix End of Series 1
A number of times people have asked about my desk over the years, and I'm curious as to why! I've visited other tech companies in the bay area, and the desks I see (including for 7-figure salary engineers) are the same as everyone else, in open office layouts. At Netflix it's been open office desks, and all engineers have the same desk.

Does some companies give bigger desks for certain staff, or offices, or is it a country thing (Europe?).

brendangregg··on Netflix End of Series 1
I've been clear on HN multiple times that I saw no future in the Solaris/SmartOS ecosystem by 2014, which is one (but not the only) reason I left it. I was right too: https://www.itjobswatch.co.uk/jobs/uk/solaris.do
brendangregg··on Netflix End of Series 1
I touch it now and then; search for "BSD" on https://www.brendangregg.com.

Netflix has hired more great BSD folk, so I haven't needed to do a lot on it for a while. I do enjoy it though, and it's usually much easier to read and understand the kernel code.

Most recently I've been talking to folk about eBPF on BSD. Maybe I should do a blog post about it (force me to go talk to everyone again and get their updates).

brendangregg··on Linus on Line Breaks (2020)
Just a note: books, magazines, slide decks (and more*) aren't getting any wider. As time goes on, and code and command output gets wider, it's getting a bit harder to write educational materials. I have slide decks and books that have "here's the left hand side of iostat -x" then "here's the right hand side." With the latest sysstat/iostat version, that's only getting worse (it may need three parts). That's fine if it's truly necessary.

When I develop new tools, I try to keep the default output within 80 chars if possible -- not because of terminals -- but because of all the other locations the output may appear.

* also for jira tickets, github threads, random blog posts, etc., for the output to stay on screen without fiddling with a horizontal scrollbar.

brendangregg··on Tracing SSL/TLS connections using eBPF
We use bpftrace to feed GUIs at Netflix (json output was added so that it can be incorporated in GUIs). You can come up with situations where bcc is better, but you should really make that clear in the article so as to not mislead beginners.
brendangregg··on Tracing SSL/TLS connections using eBPF
People learning to code observability tools in eBPF should be starting with bpftrace, and the tutorials should too. I've made this point in blog posts and keynotes etc. Doing it using bcc is how we did it 6 years ago -- it's out of date.
brendangregg··on Tracing SSL/TLS connections using eBPF
Thanks for your comment. Yes, it's a bummer to see people waste time reinventing something, and, to then have multiple solutions to the same problem. It's clutter, and wastes end user's time to sort through.

One solution is to spend several years developing expertise and finding the edge of the known world, beyond which is yet to be explored. In this space it means knowing all the tools that are out there. It's no coincidence that I find it easy to draw these diagrams:

https://www.brendangregg.com/Perf/linux_observability_tools....

... and to find new areas where there are missing tools. These skills are connected!

But I'm lucky to have had roles where I have spent over a decade specializing in performance analysis and tooling. For a lot of people with limited time, your best bet is to not learn the entire world, but to learn some narrow specific use case and see what tools exist just for that, and how it can be improved. Just like you say: a very specific use case.

Plus check the open issues on bcc/bpftrace github, since there's stuff there (not just bugs, but tool/feature ideas) that haven't been done yet.

brendangregg··on Tracing SSL/TLS connections using eBPF
bcc comes with the sslsniff(8) tool that does just this, created by Adrian Lopez in 2016. I don't know why the article didn't mention that, since they are using bcc.

It's also a trivial bpftrace program, which we've been recommending for beginners over bcc for years (including in my eBPF summit keynote two years running).

This article would make sense if it was written in 2015.

brendangregg··on You don't need to burn off your fingertips (and other biometric myths)
For anyone thinking of using fingerprints: There's a number of people, especially some older women (due to a lifetime of hand washing dishes) and chemotherapy survivors, who no longer have legible fingerprints. Insisting all users authenticate with legible fingerprints becomes discriminatory.

I'm not saying fingerprints are bad, just that there must be a process to support those that do not have them.

brendangregg··on ZFS is mysteriously eating my CPU
Systems Performance 1st edition sold over 10k copies for the English edition, and that was half Solaris when it was falling in market share. 2nd edition is more Linux focused, and shows how eBPF fits into the toolset, so I'd assume it'll sell more than 10k. (The publisher has been changing their author's portal and I haven't been able to login for months now, so I don't know the current numbers for 2nd Edition.) Various companies make it required or recommended reading for new engineers, which helps a lot (thanks!).

How much pirated PDFs hurt these books I'm not sure, maybe a little, maybe a lot. With my BPF book, a "rough cut" was published on Safari and then that did the rounds as the PDF (and still does) even though it was unfinished and buggy. That's what annoyed me the most about it, was that people were reading a broken version and may be thinking it's the final version.

Thanks for getting and recommending it!

brendangregg··on ZFS is mysteriously eating my CPU
Most issues we hit are solved by the monitoring GUIs (Atlas, FlameCommander, PerfDash, S3 log collection, etc.), but some aren't, and there's a ton more information available from the CLI tools. Imagine losing your five nines because no one can SSH on and try a few ad hoc tools.

But yes, CLI can be dangerous, even for observability. E.g., people running strace(1) on production apps and causing outages due to the strace overhead (I wrote a prior post about that). You need to understand the risks and overheads of all tools. It's why I have a "pull no punches" policy when writing eBPF tool man pages [0]: If the overhead can be bad, it should say so clearly.

[0] https://github.com/iovisor/bcc/blob/master/CONTRIBUTING-SCRI...

← PreviousPage 2 of 16Next →