According to the instructions on the site, you're supposed to load the entire package (all icons) onto your site which is over 80KB. Is there a subsetting option? I use Fontello where you select the icons you intend to use and download a custom font only containing those icons.
http://icomoon.io/ allows you to build an icon font using font awesome characters. Not sure if they have the most recent version of font awesome on there though...
While not exactly what you are asking, try to use the CDN:
<link href="//maxcdn.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css" rel="stylesheet">
If more websites are using it chances are that your user has the css already in their cache.
I already use Fontello to build custom icon packages (and most of my icons are from Font Awesome I think) :)