Is there an authoritative beginners guide to linux performance analysis? A cursory glance at trace-cmd tutorials shows I have a lot to learn. Or maybe it would be more beneficial to start playing with a tool like sysdig?
Is there an authoritative beginners guide to linux performance analysis? A cursory glance at trace-cmd tutorials shows I have a lot to learn. Or maybe it would be more beneficial to start playing with a tool like sysdig?
As for learning how to do all the stuff... If you're a programmer, a good approach is to write your own simple poor performance programs to anaylze. Eg:
- burn CPU in a function
- do too much disk I/O in a function
- do lots of large size network I/O from a function
- do heavy memory allocation from a function
Now analyze these using the Linux toolset. Quantify the behavior (how much per second), show its effect on system resources (%utilized, or IOPS, or whatever), then drill down and identify the code path responsible: the function.This sounds easy, since you wrote the program to start with, and already know the answer. But there's often a lot of quirky behavior with perf tools, and assumed knowledge to learn, that you want the target to be as easy as possible.
A mistake beginners make is to aim performance tools at complex production workloads, and then get lost and confused. It's like trying to run before knowing how to walk.
I think Julia Evans is going to cover this approach in an upcoming talk, so that should become a good reference.
You'll also want to develop a good understanding of software, systems, and kernels, so that you can reason about performance. That may take years to pick up, and the study of various text books. I tried to summarize all the content in my last book, systems performance.
Edit: formatting.
Not a ton on ftrace in it, though. Maybe we'll see a new book more focused on Linux specifically with some ftrace stuff?
I've been thinking of how best to cover my new ftrace and perf_events content in book form...