So this has happened at more than one company that I've been at, only difference is that these were AWS keys and used to mine bitcoin. AWS was actually pretty good about it, we rotated the keys as quickly as we could and they dropped all the charges.