Soon afterwards, however, we started receiving reports of our users not being able to refund or charge credit cards. All that information should have been in the DB, encrypted! We quickly discovered that, on occasion, the encrypted data getting corrupted. Immediately we did what every engineer would do in our place - blame the previous engineer's code, then try and find the bug that would prove our theory right. After days of studying source code and testing theories, nothing explained occassional corruption.
Eventually we traced the beginning of our problems back to our server/framework upgrade, and found a Backwards Incompatible Change: invalid unicode code points would now be silently dropped, rather than being allowed. It turns out that all of our credit card numbers were being encrypted properly, but then DECODED using the UTF-8 Encoding and stored in an NVARCHAR column in the DB! Everything was fine in .NET 1.1 (and SQL Server 2000) but .NET 2.0 silently drops the invalid UTF-8 code points. With those code points missing, it was impossible to decrypt the data and do anything with it.
... I suppose that makes it more secure though, so there's that ...
We felt that .NET 2.0 was a big enough upgrade that it was worth adding some new warts to our system. The final hack: we found an unused pc and built a .NET 1.1 web service with two functions: encrypt/decrypt. We store credit card numbers in the database in plain text, make a call to this webservice with the row id, and it encrypts the data. This solution lasted almost 5 years before our boss accepted the pain of an hour of down time and we exported/decrypted/encrypted/imported the entire db.