If your project ever makes any money or gains any recognition, then your (ex)employer is very likely to sue.

Most employment contracts include a clause saying that "all" work you perform during your employment is owned by them. Unrelated hobby work is generally Ok. But from what you describe, under most jurisdictions your work will be deemed to be owned by your employer.

Will be very interesting to see what others suggest. But my advice is: "Don't go there!".