For browsers that don't support WOFF, just specify a browser-safe font in your CSS. Those browsers will ignore the Woff declaration and use the browser-safe font instead.
EDIT: As far as file size goes, I use two font weights on my site: Source Sans Pro regular and bold. Each font weighs 27kb - so 54kb in total. This is much less than many Javascript libraries and third-party embeds.
Here's my site (hope it's fine to link) - it's admittedly very simple and has no Javascript (although Javascript is needed to load the fonts!). In the end you have to weigh up the balance between page weight and performance for your own site.