At that moment, we can't recover from that and players connected to one of the crashed exchanges (1 server process: N exchanges) are told to restart levels, which is not an ideal outcome but it's essentially free. I hope to eventually be able to do something like use our persisted-out-of-memory (e.g. to NSQ) tapes of the orders, bring up the crashed exchanges, and fire every order back into it in order, which (since it's a stock exchange) should result in the same orderbook it crashed with minus whatever we lost during the crash.

After we can do that reliably, then I will strongly consider making an advanced mode level where a crash / market reset / rollback of trades is a planned feature of the level (they are absolutely a thing which happens in real life) and dealing with it from the player's perspective is the main task. (e.g. What would you do if you got a message from the stock exchange saying, basically, "We lost data. How much? Umm, not sure, the last orderId we have is X. Orders after X are no longer on our books but, um, I guess there exists the possibility that they will nonetheless settle successfully if they matched before we went down. ... Good luck to you!")