Evolutionary algorithms may work ok here, but I have not seen any high-performance ML tasks done with evolutionary training of neural networks in a long time. It's all backprop and, increasingly, reinforcement learning these days.

Have you ever looked at https://github.com/karpathy/reinforcejs ? Karpathy already has demos similar to yours which you could build on. The trained agents might be better.