The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
4,953 karma · joined September 27, 2011
The pgbench docs make it sound microbenchmark-y by describing making the same call over and over. If people find that this simulates actual production workloads, then yes, it can be considered a macro-benchmark.
While it might call itself a benchmark, it behaves very microbenchmark-y.
The other numbers I and others have shared have been from actual production workloads. Not a simple program that tests same sequence of commands over and over.
Helping people find ~30-3000% perf wins, helping debugging and automated bug reports, is huge. For some sites it may be like 300 steps forward, one step back. But it's also not the end of the road here. Might we go back to frame pointer ommision one day by default if some other emerging stack walkers work well in the future for all use cases? It's a lot of ifs and many years away, and assumes a lot of engineering work continues to be invested for recoving a gain that's usually less than 1%, but anythings possible.
There's a couple of problems with an apt reinstall. One is that people often don't work on performance until the system is melting down -- many times I've been handed an issue where apt is dreadfully slow due to the system's performance issue and just installing a single package can take several minutes -- imagine reinstalling everything, it could turn the outage into over an hour! The other is I'd worry that reinstalling everything introduces so many changes (updating library versions) that the problem could change and you'd have no idea which package update changed it. If there was such an apt reinstall command, I know of large sites (with experience with frame pointer overheads) that would run it and then build their BaseAMI so that it was the default. Which is what Ubuntu is doing anyway.
I also have other air quality meters. (I collect measuring devices.) I wish there was a do-it-all air meter.
I then felt it best to post a video of the bird knocking so people didn't think I was crazy. They knock with their beak: tap tap tap. Gets annoying when they do it at 6am to wake you up.
Recently I have noticed cockatoos raiding the bins in Sydney, it's definitely a thing.
1. FaaS
2. Containers
3. Lightweight VMs (e.g., Firecracker)
4. Bare-metal instances
As (and if) an application grows, it migrates to platforms with greater performance and observability.The ship has sailed on neighbor detection BTW. There's so many ways to know you're a VM with neighbors that disabling PMCs for that reason alone doesn't make sense.
- PEBS (Precise/Processor event based sampling, so that we can accurately get instruction pointers on PMC events)
- uncore PMCs (in a safe manner)
- LBR (last branch record, to aid stack walking)
- BTS (branch trace store, " ")
- Processor trace (for cycle traces)
Processor trace may be the final boss. We've got through level 1, PMCs, now onto PEBS and beyond.Glad to see "overhead" mentioned and quantified. I'd put the 2-10% at the top though, as that's heavy handed for some environments (can trigger a production fail-over).
I see magic-trace has implemented what some call "flame charts" (time on the x-axis) and not "flame graphs" (alphabet on the x-axis). The best tools do both (e.g., TraceCompass). Please do both! Will make seeing the big picture easy (flame graphs) and then zooming into time-based patterns easy too (flame charts).
Maybe related, but we once went around the Netflix parking lot with Blippar to look at the KBB value of the cars, and found that I was driving the cheapest car at Netflix (at least until we got sick of looking). Mine was a 2001 Toyota Prius, KBB value ~$1800. Not the cheapest car to ever park at Netflix, just the cheapest on the day!
We practiced on baseball diamonds, and played league games on the Google soccer field.
When I ran the BPF tutorial at USENIX LISA 2019, I had one finger in a splint from breaking it playing cricket. :-)
Does some companies give bigger desks for certain staff, or offices, or is it a country thing (Europe?).
Netflix has hired more great BSD folk, so I haven't needed to do a lot on it for a while. I do enjoy it though, and it's usually much easier to read and understand the kernel code.
Most recently I've been talking to folk about eBPF on BSD. Maybe I should do a blog post about it (force me to go talk to everyone again and get their updates).
When I develop new tools, I try to keep the default output within 80 chars if possible -- not because of terminals -- but because of all the other locations the output may appear.
* also for jira tickets, github threads, random blog posts, etc., for the output to stay on screen without fiddling with a horizontal scrollbar.
One solution is to spend several years developing expertise and finding the edge of the known world, beyond which is yet to be explored. In this space it means knowing all the tools that are out there. It's no coincidence that I find it easy to draw these diagrams:
https://www.brendangregg.com/Perf/linux_observability_tools....
... and to find new areas where there are missing tools. These skills are connected!
But I'm lucky to have had roles where I have spent over a decade specializing in performance analysis and tooling. For a lot of people with limited time, your best bet is to not learn the entire world, but to learn some narrow specific use case and see what tools exist just for that, and how it can be improved. Just like you say: a very specific use case.
Plus check the open issues on bcc/bpftrace github, since there's stuff there (not just bugs, but tool/feature ideas) that haven't been done yet.
It's also a trivial bpftrace program, which we've been recommending for beginners over bcc for years (including in my eBPF summit keynote two years running).
This article would make sense if it was written in 2015.
I'm not saying fingerprints are bad, just that there must be a process to support those that do not have them.
How much pirated PDFs hurt these books I'm not sure, maybe a little, maybe a lot. With my BPF book, a "rough cut" was published on Safari and then that did the rounds as the PDF (and still does) even though it was unfinished and buggy. That's what annoyed me the most about it, was that people were reading a broken version and may be thinking it's the final version.
Thanks for getting and recommending it!
But yes, CLI can be dangerous, even for observability. E.g., people running strace(1) on production apps and causing outages due to the strace overhead (I wrote a prior post about that). You need to understand the risks and overheads of all tools. It's why I have a "pull no punches" policy when writing eBPF tool man pages [0]: If the overhead can be bad, it should say so clearly.
[0] https://github.com/iovisor/bcc/blob/master/CONTRIBUTING-SCRI...