I would add a counter argument to "it feels hacky" because to me, it feels like the perfect solution: icons are symbols that we use to explain things, just like words.
The font format in particular seems perfect for this:
- One single download to get your entire icon set
- Font rendering engines have been around long before web browsers, so they are relatively mature on all platforms
- Scales just as nicely as SVG