Mystery of the Melting Servers, solved.
Mystery of the Melting Servers, solved.
Once code is no longer synchronous, and tasks are being juggled around and swapped between, it becomes virtually impossible to predict what might cause this sort of bug.
Just because you can't determine the exact cause of the leak, or even detect all of them, doesn't mean that having some indication of memory usage initially and under test loads of different (reproducible) amounts at different intervals and comparing them across builds can't get you a hell of a lot of utility.
Er, sort of. Integration and e2e tests often catch problems nobody specifically, explicitly envisioned. That's part of what makes them so valuable! Right?
Sort of like if you are planning to drive some car you recently did a bunch of work yourself to on a long trip. You've probably already tested all the things you think might be wrong, but that doesn't mean it's not worth taking it for a drive for an hour around town and on the freeway to see if the unexpected happens, so you can deal with it close to home. You're testing the car, just in a vague "shake it and see what falls out" way. You are specifically testing for that, it's just not guaranteed to find any or all problems. It's still better than nothing most times.
In answer to the original question, I stand behind my answer with 100% confidence. The test infrastructures I build catch memory leaks all of time, and I do not consider it abnormal for them to do so. If you mean to say that I cannot be confident that I have caught all of the memory leaks, well duh, of course I can't. But that's not what was asked, and that's not what I said.
To expand on what was actually asked, though, whether a framework catches all, some, or no memory leaks is irrelevant. Because if a team is at least testing for it, I'll bet they're just a bit more rigorous in their coding than a team that just deploys to production and waits to see what breaks.
Could you elaborate on that, please? I'm curious. Thank you in advance!
But in general, no matter the project, you need a few key pieces:
1. A test suite that thoroughly exercises the code. You'll need this for...
2. A tool such as valgrind to watch memory as your tests in #1 run. Build time, ad hoc, stick in the pipeline anywhere you like.
3. A static analyzer to take a first pass at the code and say, "hmm, this might leak/crash/kill puppies." So, yeah, the order is off as this should be the first thing you do. It can run on your dev machine or on the build machine. Expect lots of false positives.
(optional) 4. A code coverage tool to make sure you're not missing key pieces of code that need exercise.
Assuming your module's interface isn't too complicated, it shouldn't be terribly difficult to write a test fixture that fuzzes a module, and verifies that its memory consumption does not go outside a certain bound.
The first round of bugs you'll catch might be design flaws more than true memory leaks - places where the module isn't designed to limit its own memory usage in the face of adverse (or adversarial) input.
Get those pinned down, and then you can use it to catch memory leaks with confidence. Let it run for less time if you're only worried about catching fast leaks, and for more time if you need to catch the slow ones, too.
It won't catch all possible errors but it will catch most. On the plus side it will only nag about actual errors, not potential errors like "safe languages".
“That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it for hours.”
That’s exactly the problem that thorough automated test suites try to solve. When you have a robust mix of unit, integration, end-to-end and performance tests, you can greatly reduce the number of unintended regressions that make it into production. Not eliminate them completely, but definitely reduce them a tonne.