Profiling Go Applications with Flamegraphs | Hacker News Reader

Profiling Go Applications with Flamegraphs

brendanjryan.com

152 pointsingve27 comments

A nice writeup, thanks. There are a few variations on this workflow that I've found useful in practice; perhaps they'll be helpful to some folks:

- Linux perf can profile unmodified Go programs. This is handy when your application doesn't expose the /debug/pprof endpoint. (http://brendangregg.com/FlameGraphs/cpuflamegraphs.html#perf has detailed instructions)

- Recent versions of https://github.com/google/pprof include a flamegraph viewer in the web UI. This is handy when you want a line-level flamegraph instead of a function-level flamegraph.

Does anyone else have this problem with go indentation:

    func (sc *SimpleClient) Timing(s string, d time.Duration, sampleRate float64,
        tags map[string]string) error {
        return sc.send([...]

The wrapped parameter list is indented to the same level as the function body. Is this how it's supposed to be done?

Surprisingly, it is. (Though personally in Go I wouldn't wrap the parameter list at all (and I think maybe most also doesn't?). Then you just let your editor soft-wrap to match current window width; the continued line indent can be configured per personal taste.)

No problem with that indenting here (personal opinion), I guess makes it easier to read if your editor doesn't have soft-wraps. However, generally, if the parameters list overflow to another line like that, it becomes a good candidate for refactoring!

Yeah, it could be a new struct, especially if another parameter or two comes along (though the simplest thing has its benefits, too).

I would have liked to have seen how the author measured the hot spots in the graph. It wasn't immediately obvious to me that the `WriteToUDP` function was taking up a significant portion of the call.

Another thing that is omitted is what type of buffer was used. I'm assuming it was a `[]string` slice but why not use a `sync.Pool`?

I’m trying to figure out why I find Go so hard to read. Am I mostly alone with this?

It’s considered idiomatic in Go to use very short variable names [1], which seems masochistic to me. Every time I look at Go code, it’s like I’m back in high school using Borland Turbo C all over again.

[1] https://github.com/golang/go/wiki/CodeReviewComments#variabl...

I came here to say the same thing, and I'm so relieved I'm not the only one. Looking at those one- and two-letter variable names constantly leaves me scratching my head and wasting time figuring out what they mean.

I find it really depends.

Small variable names for iteration variables and the `this` equivalent in struct methods are fine.

As for other situations... I guess it's an art?

I've mostly interpreted that to use single character names for method receivers and loop variables. I think those are defensible because you know where to find them if you need their definition (in the method definition or in the loop).

I don't think that assigning terse names (such as the variable "ra" from the OP) to variables that might be declared anywhere in a function is helpful.

I agree, and the link I posted states as much.

I think the problem comes when people take that advice without nuance and think it gives them carte blanche to make everything as obscure as possible.

And really, it’s not about whether or not we can understand our own code, but can the future developers who have to maintain it after we leave.

I kinda worry that the Go community is creating a lot of unmaintainable code right now.

I see an intrinsic pattern to variable naming in Go and not some randomly assigned single letters.

I work on Java code where variables name look like 'reconnectDelayToInitiallyEstablishJMSConnection" Even though very clear name it really exhaust me while reading code like this. Java explicitness things like spreading code over dozens of files and directories for a functionality that could ideally be in 1-2 reasonably sized files. And methods that actually do something instead of calling another methods. So I guess code I deal with is understandable at a method level which finally does something. But overall it is too sprawling to fit everything in mind while looking at a functionality.

I would just argue for a happy median. Enough to convey intent, without being verbose. Your java example is clearly exhausingly verbose (I’ve seen similar in C#), where it seems in Go it would probably be written:

rd := 1000

How is anyone supposed to understand at a glance what this value is for? Why not just:

reconnectionDelay := 1000

Intent is clear.

And now, I have strayed way too far off topic (I’ve done some flamegraph style debugging in .Net, super useful!), and should probably apologize to the OP.

(to continue straying for a moment..) perhaps though it best depends on the variable specifics: like how often it appears, how far apart multiple uses are, etc.

Some natural limits might be: (1) single letters for extremely local terms whose structural meaning is more salient than denotation (canonical example: loop counter); (2) fully spelled-out terms for globally-significant terms (not necessarily in global scope) whose denotation is crucial (canonical example: an app configuration value).

Even two short camel-cased words often seem unnecessarily verbose for a loop counter to me. Whereas an important global config variable might justify the full THIS_IS_WHAT_I_AM_FOR treatment.

Java went through a phase of not having inner classes, autoboxing, enums, generics, and looping over iterables. The code from that era was burned to the ground for being unreadable shit.

This is survivable, if they're motivated to fix the glaring omissions in Go.

Though that would still be idiomatic go code. For what it is worth, I do agree with you. Short variable names on a function are a hassle. You have to lookup what they mean, which slows down understanding.

n = copy(p, b.buf[b.r:b.w])

Oh better go to the top and see what those are again...

https://github.com/golang/go/blob/master/src/bufio/bufio.go#...

I find to understand a piece of code, I need to know what all the involved variables are, which I'll have to do regardless of long or short names.

Once I know what they all are, I find it's easier for me to track them through the function if they have short names.

Something java style like

    numBytesWritten = copy(destinationBuffer, reader.buffer[reader.readPosition:reader.writePosition]

is much harder for me to follow.

I didn't look at the file but I find that perfectly comprehensible. I can see read & write pointers in a buffer, n as bytes copied. Not sure if p is source or dest w/out looking but my guess is dest?

Maybe it's a cultural thing - lots of C code looks like this from Lion's UNIX book onwards. I still find this style easier on the eyes when it's clear from context what is going on.

For i/o code this is about as obscure as using "i" for a loop index.

There's a great rule of thumb coined by Andrew Gerrand: "The greater the distance between a name's declaration and its uses, the longer the name should be." [1]

There's no reason to use any more than a single character for a loop variable. Who cares how old the convention is?

[1] https://talks.golang.org/2014/names.slide

I find Go quite easy to read. Maybe you just need a bit more time with it? It does tend to prefer short variable names, which my brain 'likes' -- I think some people have an easier time with shorter names, some people with longer names.

There must be individual differences at play. I find Go the easiest to read of any language I know (and I've only been looking into it, reading more than writing, for a few weeks). This is part intrinsic limitations (it's a simple language, and there tends to be one obvious way to do ordinary things), in part culture (a strong leaning towards consistent idioms). Perhaps these aspects of the language are a more natural fit with some of us than others.

As for short variable names -- I think they're a readability boon where they are essentially placeholders into common structures that you can read as a whole unit at a glance (counters in loops, readers/writers/buffers in input/output idioms etc), but detrimental elsewhere.

How are flame graphs a better way of displaying profile data than viewers like paraprof [1] or cube [2], which have tree/flat and inclusive/exclusive views, as opposed to essentially just an inclusive tree view? Consider seeing the effect of something called all over the place in a complex system.

[1] http://www.vi-hps.org/upload/material/tw-score-p/vi-hps-tw-s... [2] http://www.vi-hps.org/upload/material/tw-score-p/vi-hps-tw-s...

Flamegraphs are good for CPU profiling (as far as I've used them), I've used them to debug an issue when one of our programs was "consistently" consuming high CPU. As the "stack" was mostly on CPU, during sampling, this stack was picked up most of the time, hence in flamegraphs, it was represented with a relatively longer bar.

But do they help to debug issues related to CPU spikes? I have my reservations because they are generated using sampling, and flamegraphs won't give a clear insight into who is causing the spikes.

One way to solve this I can think of is, if cpu spikes are deterministic, then we can take 2 profiles: 1. span consisting of no spikes 2. span consisting of at least 1 spike

And then "diff" them to get stack causing spikes.

Is this the correct way?

In general, how do people solve CPU spikes issue?

Whenever I see something like this I think of the 2015 video netflix[0] gave also using flamegraphs but for node.

Personally I want something like this for py.

In searching for the vid I see netflix also have a medium post(2014)[1] about it.

[0]: https://www.youtube.com/watch?v=O1YP8QP9gLA

[1]: https://medium.com/netflix-techblog/node-js-in-flames-ddd073...

Video of this in action: https://www.youtube.com/watch?v=aAhNDgEZj_U

Reply on news.ycombinator.com