I've worked with hardware accelerated vector graphics engines (2D GPUs). It sounds like a good idea to do everything vector, because some things are very simple while retaining quality as you said. But in my experience, the vector operations are so expensive compared to raster ones that you go back to rasters to maintain framerate.