I've been working on a warehouse management software (that was running on mobile barcode scanners each warehouse worker had, as he moved stuff around the warehouse and confirmed each step with the system by reading barcodes on shelves and products).
We had a test mode, running on a test database, and production mode, running on the production database, and you could switch between them in a menu during the startup.
During testing/training users were running on the test database, then we intended to switch the devices to production mode permanently, so that the startup menu wouldn't show.
A few devices weren't switched for some reason (I suspect they were lost when we did the switch and found later), and on these devices the startup menu remained active.
Users were randomly taking devices each day in the morning, and most of them knew to choose "production" when the menu was showing. Some didn't, and were choosing the first option instead.
We started getting some small inaccuracies on the production database. People were directed by the system to take 100 units of X from the shelf Y, but there was only 90 units there. We looked at the logs on the (production) database, and on the application server, but everything looked fine.
We were suspecting someone might just be stealing, but later we found examples where there was more stuff in reality on some shelves than in the system.
At that time we introduced a big change to pathfinding, and we thought the system was directing users to put products in the wrong places. Mostly we were trying to confirm that this was the cause of the bugs.
Finally we found the reason by deploying a change to the thin client software running on the mobile devices to gather log files from all the mobile devices and send to server.