Yep, that's why I am planning on getting rid of them completely. I did a quick test this morning, seems like removing references to the fonts from the theme and adjusting some font weights should do the trick.
I use type subsetting and only serve about 50kb worth of fonts (4 variants in total). 300kb seems excessive.