Hacker News Reader

Comment by brendangregg | Hacker News Reader

BPF (formally eBPF) provides the programmable tracing capabilities that Linux previously lacked. Things like custom latency histograms. It works with other built in features: kprobes, uprobes, tracepoints, etc.

So yes, BPF is becoming the standard and the future. It made it into the kernel (well, the enhancements needed for tracing did; BPF has been around for a long time as just a packet filter). In other words, BPF won. bcc is likely the future, but we'll see other front ends as well (eg, ply).

View on HN

I should add that Linux has two tracing front ends built in: ftrace and perf_events. And perf_events is building BPF support (see lkml). bcc is another front end for BPF. If this is confusing, I should categorize them like this:

back end technologies: ftrace, kprobes, uprobes, tracepoints, BPF

front ends: ftrace, perf_events, bcc

ftrace is hard to categorize: it is a collection of both back end and front end capabilities (although the front end is /sys). Also, it's being renamed to just "trace".

Thanks Brendan! A more navel-gazey question I believe you're well positioned to answer:

For a long time, there seemed to be a conventional wisdom that Linux tracing was mediocre, because it doesn't have DTrace. Has Linux approached or passed the "caught up" point?

It hasn't caught up yet, but it's getting pretty close. Although I'd think about it in terms of three dimensions:

- technical capability: we should pass it this year. We'll be technical capable of solving pretty much the same issues and then some.

- ease of use for script developers: bcc is currently ~10x more verbose to program. ply is much easier, but that project is in its early stages. This won't matter that much for Netflix, since much of the time staff will use this from a GUI (eg, Vector). It also won't matter that much for people who use the many provided tools, like opensnoop, tcpconnect, ext4slower, etc. It matters most for whoever codes the GUI and those tools (eg, me).

- marketing, awareness, brand: Sun marketing, sales, trainers, and evangelists, did an awesome job - everyone knows about DTrace, even if they've never used it. In comparison, bcc/BPF (and many other Linux kernel technologies, like ftrace) have virtually zero marketing budget or professional marketing staff promoting them. Awareness will take the longest to catch up.

I also summarized the current state of bcc/BPF in http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-ro...

... I should add, a tipping point for awareness may be when a commercial performance monitoring company adds BPF support (and latency heat maps, etc), and then promotes this feature. There's a lot of performance monitoring companies out there, many are well funded, and have professional marketing/sales departments.

New account, just to ask this.

I understand how much effort is involved in writing / publishing.

Are you planning a Second edition of your "Systems Performance, Enterprise and the Cloud" text, especially in light of the pace of development in the Linux performance / profiling space?

Thank you for all your countless contributions.

Yes, I'll be writing more books, more details later; as for a 2nd edition, well, the 1st edition is probably more useful in the meantime than you might think. Lets say you install bcc/BPF tomorrow. Ok, now what? The hardest part is to know what to do with these superpowers. Apart from the provided scripts in bcc, the sysperf book has other DTrace examples that you might need, and can then port over. The hardest part of DTrace was knowing what to do with it, so all the prior DTrace documentation will be good material to browse and be inspired from, including my DTrace book. Many of my bcc/tools originated as versions I wrote using DTrace.

Reply on news.ycombinator.com