Not only you can, but in many (if not most) cases you should - because of performance and security benefits. The drawback is your site will no longer use the built-in WP search functions, but there are ways around that.
Couldn't you put WP behind a reverse proxy and cache the hell out of it?
We cache the hell out of our WP marketing site. Cloudfront > Varnish > Apache > OPCache > MySQL Query Cache. It feels fast to visitors. People in the admin area still suffer.