SVG is kind-of terrible for maps, but you can get pretty small with GeoPackage (read: sqlite). I recently spent a bit too long on exactly this problem and ended up with the following.

116KB - 5MB for country borders

16MB - 52MB for ~50K city/county level borders based on geoBoundaries

The range of sizes depends on how much custom compression/simplification you put into it. The source files are about 10x bigger, but that's already pretty small.

Topojson might be even smaller though.

Check the repo for details /selfplug https://github.com/SmilyOrg/tinygpkg-data