But those artifacts have usually faded with time and exposure to the light, so even those don't show you what the colors actually looked like, unless they have been sealed away from light exposure all this time.
Also, we still have the techniques and materials used to make old paints. New batches can be whipped up (assuming they aren't too toxic) for comparison. Lead based paints obviously are more problematic, but modern technology is really good at replicating paint colors.