You shouldn't redirect for assets that dont exist, even if you do for pages...
At first we would only treat urls with certain file types (jpg, png, mp4, etc) as files when found on the server, the rest as pages and we switched from a whitelist of file types to anything that had an asset directory path in the url. Our original htaccess rules were definitely deficient. The idea was that we wanted tracking pixels and the like to be able to run code.