Hacker News Reader

Comment by haberman | Hacker News Reader

Argh. This talk reinforced my existing view of Linux tracing: it's really fragmented.

Five years ago I tried to make some sense of this by researching all of the existing technologies. In the kernel I found:

   - ftrace (https://lwn.net/Articles/290277/)
   - tracepoints (https://www.kernel.org/doc/Documentation/trace/tracepoints.txt)
   - kprobes (https://www.kernel.org/doc/Documentation/kprobes.txt)
   - events (https://www.kernel.org/doc/Documentation/trace/events.txt)

Now apparently we can add:

   - BPF, a packet filter that grew into a tracing framework
     (https://lwn.net/Articles/599755/)

In user-space we have:

   - perf
   - systemtap
   - lttng
   - other, random, fragmented things

This talk seems to add a bunch of other fragmented user-space tools.

I don't mean to put down anybody's work, but this stuff will never be user-friendly as long as it remains so fragmented, IMHO.

View on HN

I'm of a similar mind. However, somewhat recently I came across this article which helped provide a framework to think about all these things. Turns it that it's not just a flat space of competing tools:

https://jvns.ca/blog/2017/07/05/linux-tracing-systems/

Excellent post, thanks!

It's not that bad.

- Some of those are not in-tree, like LTTng and SystemTap.

- Tracepoints, kprobes, events, and uprobes are all event libraries used by perf or ftrace, just like DTrace had multiple providers (fbt, pid, etc).

The real fragmentation is perf and ftrace, since both are in-tree front ends. That's not too bad, and they both have different strengths.

eBPF is weird in that it's neither an event library or a front end. It's programmatic capabilities. We're mostly using an out of tree project, bcc, to run it.

What's fragmented about it? Almost everything he showed there was a script that use ftrace, kprobes, or BPF to measure something specific. Since those are all available in the kernel at the same time, you can certainly think of them as a single API.

So really it's a branding problem?

I wouldn't go that far; there are important differences between them.

I think lttng has kernel tracing. I don't know why the fragmentation you describe is "bad", though; it really depends on the tools themselves.

If there's just a variety of tools for the same task, then that's healthy competition and how you get better software.

If no single tool can fulfill all your tracing needs, that's still not necessarily a condemnation of the tools. It's entirely possible that each tool can complete a subset of tasks, but is significantly simpler to use as a result, so SUM(effort to learn tools you need) may still be comparable to the effort of a theoretical omni-tool.

He didn't mention this in this snippet, but the BCC (BPF Compiler Collection) intends to make this much simpler[1]. In particular it lets you write a tracer in Python (with the BPF program written in C) that attaches the BPF program to whatever types of probe points you like. So while internally there might be all this fragmentation a user shouldn't have to deal with it as much.

[1]: https://github.com/iovisor/bcc

Brendan used to be Mr. DTrace User. (Not Mr. DTrace -- that was bmc, ahl, and mws.) But the world isn't using Solaris or FreeBSD, so I guess he moved on like most of the rest of us Solaris diaspora. Still, every time I see one of Brendan's blogs I know, deep down, he must miss DTrace; I sure do. This video doesn't help me feel at home with Linux, but it's a resource for when I need to trace something. Mostly though, when I have to debug something on Linux, I do it the pre-DTrace way, which is to say: the hard way.

Isn't that kinda the ethos of Linux? Every program does one thing and does it well.

No that's the UNIX philosophy. And Linux threw that out the window from day one. From file systems to solving /dev/poll and containers.

The good (and bad) thing about all of the technologies that make up "containers" on Linux is that they can be used by separate projects. Chromium uses seccomp, systemd uses namespaces and cgroups, a bunch of tools use AppArmor/SELinux.

But ultimately the reason that this is the current state is because of how Linux is developed. Trying to push something like Jails or Zones is an exercise in futility because the patchset would be too large, would touch everything, and the infrastructure would likely not be reusable by other people.

Some of those are old tools some are old. This will always be the case

Reply on news.ycombinator.com