Most Sampling Java Profilers Are Terrible
psy-lob-saw.blogspot.com
psy-lob-saw.blogspot.com
What is really helpful is that you can run DynaTrace in production without impacting performance. DynaTrace agents run on the production web/app servers and send data to DynaTrace collectors.
This makes resolving production issues way easier. You sort by CPU time in descending order and work down the list. You already have the stack trace and data associated with the request or SQL query.
They bought 20 or so licenses and never actually used them in prod - mostly because of the effort in building up a sane config for ops to use. The guy who set it up did a pretty good job, but after coming up to him and mentioning the 2hr/day stop the world GCing going on, dynatrace wasn't able to see them, where the jvm logs could.
So yeah - it's a good product, but you better hope you have the money for it and the patience to build a good config.
Now, I wouldn't typically care, but you can get at lest 95% of the same information, including yes, stack traces, with ETW for basically free - in cost and in perf impact on Windows at least. Throw the results in a flame graph and boom, done. Windows in particular has so much great inbuilt stuff for troubleshooting perf but its hard to find and not well marketed, so these other folks make a killing selling an inferior product.
Good news is that all APM vendors provide free trial versions. So - give it try if you want to see if it gives you more than ETW!
(end of commercial) :-)
The worst thing about the JVM aprofiler won't tell you about how much time the JVM spends JITing/class loading. For short running processes you might be surprised to find that the profiler tells you everything is okay but in my case the program still needed one second to start.
I was pleasantly suprised that the equivalent python programm merely needs 100ms from start to finish (essentially feeling instant) even though it should be slower overall.
For a sampling profiler to give meaningful results, it shouldn't be waiting for safepoints. Ideally it would capture stack traces very cheaply (just walk the stack frames, noting the return addresses) and turn those numbers into symbolic locations on its own time. Yes, inlining is going to confuse things - that's the downside of working with optimized code. Optimization inherently intermingles code that may be separated by some distance, and any one instruction may be a fuse of multiple source lines. If your instrumenting profiler prevents this, you're no longer measuring the actual production code.
Java makes it harder, of course, with code on a GC'd heap, dynamic codegen, etc.
http://techblog.netflix.com/2015/07/java-in-flames.html
Java mixed-mode flame graphs provide a complete visualization of CPU usage and [...] can identify all CPU consumers and issues, including those that are hidden from other profilers.
It's only when the profile starts getting pretty flat that the inaccuracies of this method start showing up, but for run of the mill performance problems where you have nice fat spikes sticking up here and there, it is perfectly adequate.
So far, only VTune has resulted in actionable profiler results (there may be others); within hours (not weeks) we had solved many issues by switching to it. A profiler is as indispensable as a compiler, don't skimp on it - you're going to end up spending that money one way or another.
Does anyone have any recommendations, preferably that can tell you (some) information at the line level. I never realised how spoilt I was with C/C++ options (gcov and valgrind aren't great, but they mostly do the job).
Yes sampling profilers are inaccurate, and most users know the limitations. It is deliberately a profiler on the one side of the trade-off.
The google paper was more interesting.
IOW the point of the blog post is not that sampling profilers are bad or that sampling profilers on the JVM are bad, but that some sampling profilers are bad because they use bad methodology. At no additional cost you could be using profilers that use a more accurate mechanism.
To me that is very useful information. I'm always looking for opportunities to get something for nothing.
I would also not characterize users of sampling profilers as knowing their limitations. Taking a WAG I would guess less than 50% understand the limitations. A much larger percentages know limitations exist, but I don't think they actually know what to do about them.