This is not an approach any other app on any platform has historically used, and it doesn't seem sustainable if every app you install has to modify your hosts file to use a hack like this to detect whether it should handle files or not.
If you want the browser to be able to give the OS a file handler and have the OS present an option to install the app if it's not installed, that should be handled at the platform level, not on the website using a hack like this.
Why can a file not simply be downloaded with a page displayed showing a link to install the app and also instructions to open the file, trusting the user will know if they already have it installed? At best, you're talking about a very small UX optimization. Emphasis on the "kinda" in "kinda important."