There isn't a permission for that though - it's all or nothing. I agree that it should be more granular; each app should really have its own scoped file storage area by default, with "access anything" being reserved for file browsers, backup software, etc.