Thank you all! Tons of great ideas, now I scratch my head and thinking on how to get them into a simple API. Here is what I think virtual display API should look like: https://github.com/pavlobu/deskreen/tree/master/drivers
Definitely seems like a helpful project. The stars on GitHub have jumped from 1 to 1,900+ in just one week! Enjoy the ride and thanks for open-sourcing your work.
Another thing is Linux, drivers for virtual screen already exist, check xvfb and xorg-video-dummy