DTrace made this really nice, because you would write your tracing scripts in a high-level, awk-like language, which is the sort of thing well-suited to the purpose.
Think of the current bcc/Python/C interface as a lightweight skin that was necessary during BPF development to kick the tires on various features, prototype tools, see what else needed to be done, etc. It may be good enough to stay around, as lots of tools have been written for it that will get used and be valuable. But there's room for higher-level languages too.
If Sasha keeps developing his "trace" tool (and its summary counterpart, argdist), that may serve many such custom needs (as another option). See the various examples: https://github.com/iovisor/bcc/blob/master/tools/trace_examp... , like:
# trace 'sys_read (arg3 > 20000) "read %d bytes", arg3'
TIME PID COMM FUNC -
05:18:23 4490 dd sys_read read 1048576 bytes
05:18:23 4490 dd sys_read read 1048576 bytes
05:18:23 4490 dd sys_read read 1048576 bytesIf we have some kind of spec (even if it's not formal), it might be possible, since they are roughly equivalent, AFAIK. However, since I haven't worked in-depth in either, I'm unsure what work would be involved: it's possible that only a subset would compile.
Anyways, this is probably a good goal to shoot for. Dtrace is the system tracer on pretty much all of the other big unixes, so it's a good idea to support as much of its language as possible. Plus, there are a lot of scripts already written in DTrace's language: having access to them would be invaluable.
But with the warning that I'm not a lawyer: before beginning work on a DTrace/BPF front-end, I'd start by talking to a copyright lawyer to see if permission or a license is needed from Oracle. DTrace is Oracle copyrighted, and re-implementing a DTrace front-end on Linux sounds a lot like re-implementing an Oracle-copyrighted API.
If I am, than all we'd have to rewrite was the compiler backend, which is much easier, so that would be nice.
% file `which dtrace`
/usr/sbin/dtrace: ELF 32-bit LSB executable 80386 Version 1, dynamically linked, stripped
The problem lies in the fact that neither the GNU/Linux kernel, nor the GNU applications provide DTrace probe points. On Solaris and thus on illumos, and thus on SmartOS, there are tens of thousands of probe points and numerous probe providers. Some external applications, like PostgreSQL or PHP, added DTrace probes, and all is well on Solaris / illumos / SmartOS. Some, like node.js had providers and probes added by engineers at Joyent.
http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/
http://dtrace.org/blogs/dap/2013/11/20/understanding-dtrace-...
http://dtrace.org/blogs/blog/category/node-js/
GNU/Linux would have to do the same thing. It currently only has but a handful of DTrace probes and providers, which is understandably not very useful.
I was talking about the DTrace language, which I have been avoiding calling D up until this point, so as to avoid confusion with the other D. In this post, when I talk about D, I will be referring to the DTrace language.
Linux now has a tracing system called eBPF, which provides many of the same advantages of DTrace. This is what Gregg's blog post was about.
However, eBPF requires compiling tracer scripts, or at least parts of the tracer scripts, into bytecode (IIRC). Currently, the bytecode is usually compiled from C code. However, it is ungainly and impractical to write a bunch of C every time you want to run a trace. So I asked if they had plans to support compiling a higher level language. At this point, somebody suggested that somebody should work on compiling the DTrace language, D, to eBPF bytecode. I thought that this would be a good idea, and we were discussing how viable it would be.
I thought that you had suggested using Illumos DTrace as a base for this compiler. Since by my impression, Illumos DTrace interprets D, I thought that this would require almost a complete rewrite, and thus it wouldn't very helpful.
It seems you meant something else. So what did you mean?
It is a really nice language, although missing a few things that BPF can do that DTrace can't (like saving and retrieving stacks), so it'd need to be enhanced. But with the warning that I'm not a lawyer: before beginning work on a DTrace/BPF front-end, I'd start by talking to a copyright lawyer to see if permission or a license is needed from Oracle.
I fail to see how Oracle would be relevant, given that the entire DTrace is released under the CDDL. My question to Brendan was why he couldn't just take or enhance the existing DTrace codebase to do the front-end for BPF?
(Personally, I think the other guy working on integrating BPF deeper into the GNU/Linux kernel, rather than biting the bullet, taking DTrace and doing the same thing Solaris engineers did to the SunOS kernel, is terribly mis-guided, especially since some probes and providers already exist in Linux. In the end, it will be a "Linux zoo", like everything else in Linux: "56" competing solutions for doing one thing, none of them comprehensive, and no consistency. Linux history is being repeated again. You have one comprehensive tool which works across several operating systems, DTrace, and Linux is yet again different from everyone else. Reminds me a lot of Microsoft Windows.)
>My question to Brendan was why he couldn't just take or enhance the existing DTrace codebase to do the front-end for BPF?
Brendan didn't answer, so I don't know, but my guess was that turning the interpreter into a compiler would require a near-complete rewrite.
Yes, D everyone would be nice, but what exactly does it mean? We can share DTrace scripts? Docs? Blog posts? Books? I've been porting them all over to bcc/BPF. Am I missing something?
People have already began work developing new languages, eg, ply https://wkz.github.io/ply/. What if we develop a language that's much better than D? We need to make enhancements, anyway.
I should reiterate something I covered in the post: most people won't care about this. Most people didn't write DTrace scripts when they had the option to (did either of you write DTrace scripts? have some on github?). Most people used the tools. And today, people can "apt install bcc-tools" and use DTraceToolkit like tools.
If someone wants to engage lawyers & Oracle and see if or what needs to be done to use DTrace, then great, it'd make my job easier when developing these tools (and I'd sell more DTrace books :). But I'd also like to see someone take a swing at developing a better language as another possibility.
Heck, maybe I'll learn some requisite knowledge and get on it.
I've been interested in learning DTrace for some time now, as it seemed like a tremdously helpful tool, but my systems are all linux, I don't run Fedora or RHEL, so systemtap is out. In recent months, eBPF has become a viable option, so I've been looking into that, but haven't had time to dive in deep yet. This makes me a poor candidate for doing this sort of work, which is why I'm not saying for sure that I'd do it. But I might be willing to try.
Don't get me wrong: I have no particular attachment to D as the tracing language. However, I do think it's worth supporting the tracing language that pretty much all of the other big unixes support if we can.
Am I less than qualified to have that opinion? Yes. Think what you will.
Since all of this requires being on a newer kernel anyway, I'd do that first, then try the existing bcc tools and see where you are at. You might discover that you can do everything you want to with the existing tools, which include some powerful ad-hoc analysis tools (trace and argdist). Or, if you're missing one or two tools and let us know, and we may develop them for you and add them to bcc.
Many work on the current Ubuntu Xenial 4.4. A few more on 4.6 (stack tracing), more on 4.7 (tracepoints), and all work on 4.9.
No language pain with DTrace, but having to know kernel structs put a damper on things like you wouldn't believe.
The problem is this: suppose you've just built mikmod on Solaris (or some variant thereof), and you're not the developer of the application. You want to see why the mikmod application's output is pausing every few seconds, causing a lag; that's all you care about - answers. DTrace to the rescue! Not so fast, you don't:
with DTrace, you know what the answer to what you want to get is - why is the output freezing every few seconds? What you don't know is what to ask to get it.
And I bet based on experience, that every tracing framework currently in existence has the same issue.
Even in scenarios where I knew exactly what to ask, correctly interpreting the results is a laborious and involved arcane art exercise in itself - you yourself wrote several books on the subject!
I think the most important thing to support would actually be DTrace's probe definition files. It would be great, for example, if the story for PostgreSQL's dtrace probe points was "your dtrace probe points are now accessible from eBPF programs too!".
https://github.com/iovisor/bcc/blob/master/examples/tracing/... https://github.com/iovisor/bcc/blob/master/examples/tracing/...
There's also a tool in /tools.
I'll whip up some PostgreSQL examples too (unless someone else beats me to it).
Is USDT expected to become the de facto standard for userspace probe points?
I did, some, in the beginning (circa 2006): I was stymied mostly by the realization that deep, deep knowledge of the kernel structures was required to make use of DTrace (I wasn't working as a kernel engineer per se at the time.)
I forgot a lot of it in the meanwhile: just yesterday I was trying to get a simple ustack() using DTrace on a running process, and I even pulled open "Solaris Performance and Tools", and eventually when the process finished, I threw up my hands in frustration. All I wanted to do was see why the running process (oggenc) was taking so long. (But this BPF thing looks far, far more complicated and convoluted than D.)
Nevertheless, I think D is ideal, because, in my case, it plays on my experience in programming AWK: apart from needing to know what in the kernel I wanted to probe, I could immediately start writing DTrace programs without having to learn the language. And that is amazing.
I'll add: as someone who has written and published countless DTrace and BPF scripts, I don't know that pursing a DTrace front end is wise right now, for reasons I've already covered.
I'm sorry you weren't able to solve that issue. I'd suggest starting with a profiler (timed sampling) if it was running on-CPU, to see where CPU time is spent.
This sounds like a legal conclusion. Care to back that statement with your real name?
And I am not yet prepared to disclose the name that the system gave me on the Internet - because Internet never, ever forgets. Roaches check in - but they don't check out.
I still don't see how Oracle is relevant in any of that, since DTrace source code is under CDDL. As far as I'm concerned, that's where the story begins and that's where it ends.
Sadly, you and I have never met. I've got questions for you, and then some.
Besides, as I understand it, it's the implementation that's copyrighted, not the interface or the idea, which is why Oracle went after Google the way they did, and why things like Freeablo and OpenMW can exist. But again, I'm not a lawyer, so this may be incorrect.
The GPL/CDDL license wars over DTrace went on for years. It was a real shame -- we should have been discussing or using a great technology instead.
There must be a hundred different ways and languages we can try, some of them may be better than DTrace's D. One of them is Oracle copyrighted, and lacks some features we need (stack saving). I'd either A) talk to a lawyer, or B) pick one of the other 99 options.
See https://github.com/iovisor/bcc/blob/master/src/lua/README.md... or examples https://github.com/iovisor/bcc/blob/master/examples/lua/trac... (this one is for tracing).
It also allows for building higher level APIs atop BCC's low-level lua integration, as opposed to writing yet another DSL.
But what would you expect for software coming out of Red Hat? Their projects are usually a mixed bag at best.