Hacker News Reader

Comment by acid__ | Hacker News Reader

It'd be funny if they ended up with bugs in their systems that only show up if the systems aren't turned off often enough.

View on HN

This reminds me of the time I "temporarily" fixed a memory leak in our servers by writing a cron job that rebooted them every 20 minutes. And then I forgot about it.

Ah memories. We have a web service in IIS which is used to generate some HTML and take a screen grab of it to generate images to display of some data.

The library was created in .net framework 2.0 and no longer exists or is maintained. It’s been on the list for 8 years to replace…

It has a memory leak which causes IIS to consume gigs of data over time, resulting in IIS completely restarting itself and causing all hosted sites to go offline for 30s. so I set the IIS pool to recycle every hour and it hasn’t caused any issues in ~ 5 years. Due to this I feel like it’s dropped even lower in priority. :(

That's the dirty pragmatic side of engineering. Technically it is fixed.

Hah, yes, I've unironically used https://github.com/kzk/unicorn-worker-killer in production before. ~900 stars on Github...

This is very likely. I worked for a company where started deploying code less frequently and we discovered a few uptime related issues.

I've seen this on services that do CI/CD, but institute a freeze for peak traffic events. In theory, the freeze serves the purpose of increasing capacity, since all services are serving traffic rather than updating software, and increasing reliability, since software isn't changing. However, it can then expose latent memory leaks that weren't visible when the services were cycling all the time.

Heroku used to cycle every dyno approximately once a day (for all I know, it still does…). Don’t ask me how I know my apps were free of memory leaks.

They still do that.

Like, say, an airliner?

https://www.theregister.com/2020/04/02/boeing_787_power_cycl...

It's imperfect code, but I actually wish full reboots were more common for critical tech like this.

How many times have we rebooted our machines this week to fix some weird issue? (or maybe my OS install is buggy, but still :P)

Lemme check:

$ uptime

19:14 up 42 days, 6:49, 11 users, load averages: 7.85 8.24 8.07

Zero times this week (or month).

That would probably be a good thing, means you can't break procedure easily!

Reply on news.ycombinator.com