Hacker News Reader

Comment by dunreith | Hacker News Reader

I led our team to a full CI/CD (Deployment) and while it felt really counterintuitive to move quickly and have the occasional bug show for our customers, the speed at which we could fix the bug has vastly offset it. Introduce a bug? Fix it same day. Introduce a breaking change? Rollback and fix it. No biggie.

View on HN

Have you had data corruptions because of the bugs that could not be rolled back quickly?

Goes without saying that you deploy to an identical test environment that duplicates the entire stack, with realistic test data. Automated testing protects from regressions, manual testing for the rest.

crickets

One (QA) can always dream!

Some stacks are quite complicated, involve various kinds of mobile devices with different app versions, custom hardware, offline functionality. It's just that not all systems are easy to replicate. But that's another problem.

This is something that I would be concerned with. If you have a multiplayer game like Runescape, and have CD on the servers, introducing a bug that causes corrupted data may be able to be patched quickly, but in that hour a lot of players could have been given items that they shouldn't have had. Then you have to distinguish which got the items properly and which didn't, and have processes set up to make those kinds of admin changes to the prod db

Fortunately not, but we're deploying the frontend for an eCommerce site so we don't deal with (much) data can could be corrupted.

Reply on news.ycombinator.com