Does anyone have good resources for creating "generative assets" like this?

I have a decent handle on js but I have no idea where to start in terms of tooling. For now, I've been trying to build basic eye tracking to apply a "block" of color over eyes with an animation in said block on top of a real-time video stream.

Elsewhere, automating mouth animations from audio / even moving a character in a random motion when audio input is provided.

Any advice where to start would be greatly appreciated!