How Basic Performance Analysis Saved Us Millions
blog.heapanalytics.com
blog.heapanalytics.com
Sometimes there's a really obvious problem that you can solve that will get you that kind of 10x performance boost. Often though you have to work your ass off for it or, even having found a problem where fixing it might even double your performance, the fix might be complex, touching multiple parts of the system.
In my experience, whilst you can often find quick wins, and sometimes these will be enough, there will always be cases where the analysis is much harder, or where more complex fixes are required.
(Disclaimer: I am a performance consultant, focussed mainly on SQL Server and .NET. My background is that years and years ago I lead the team that built Red Gate's ANTS Performance Profiler and ANTS Memory Profiler, and also worked on a number of their SQL Server tools.)
Still very useful article even if just to show how to use profiling tools, which you will need to even for much-higher-hanging fruit.
Hmm. I guess it depends on what you mean with "much". In terms of how capable our hardware is and how comparatively slow our software, yes, it really doesn't take much, there are typically orders of magnitude to be had. Sometimes that means ditching assumptions, however, and that can be hard. For example, I've often had pretty easy 10x, 100x and 1000x+ wins by removing SQL databases and associated ORMs.
Can you expand on that a little bit, please?
My understanding as a non CS, self taught programmer is that such workloads are better served by NoSql DB s. Yes, this comes with the lack of acid compliance, but spotting trends in real-time (apparently) don't require acid compliance.
Please correct me if wrong as I'm getting down voted, which is rather amusing
Current case they: have a lot of data, need a lot of compression (or low overhead as possible), need as much query-selectivity as possible (reading as few rows as possible for each query) that's why they have a unique case of selective indexes.
Bonus: mongodb agg sucks. Seach online for comparison to pg or es agg.