Hacker News Reader

You have the model play both versions of the game in the same way at the same time. The equality would of course not come from the model, it comes from your testing harness. You just use the model to automatically play the game. SO you are checking game version A(inputs) == game version B(inputs). Where the AI generates the inputs that play through the game 10 times in different ways for example.

View on HN

The testing harness would have to be updated per game feature and to accommodate model tendencies or quirks. This sounds like regression tests with llms and some extra steps.

Video games are pure functions of list[inputs] -> list[output]. Where the input are timestamped HID events and the outputs are video and audio, and in some cases stuff like rumble. The equivalence suite can treat the entire game implementation like a black box. Let AI generate millions of sequences of inputs that play through the game in various ways. Check for equality on the outputs with the same game but on the prior bevy version. Simple, very effective. Slight quirks game be dealt with as needed by allowing slight timing mismatches, slight or specifically allowed pixel differences allowed under certain traces etc.

I'm really not sure what was not clear about my statement of "Make the AI play the game pre and post bevy update. And difference down to a pixel can be flagged."

Reply on news.ycombinator.com