Related: is it possible to reliably maintain physical disk space quotas in Linux (similar to cgroups)?

Furthermore, is it possible to say how much "space" you would use if you were to create a file with a given size, accounting for block-size, fragmentation, and metadata? Matters such as block-size, inode usage, and metadata seem to make this very difficult even if you add special integration to the userspace application, for example by using stat or statfs. This could help prevent quota overruns for example.

These seem like hard problems unfortunately, and I suspect the best solution is to just create separate disk partitions for each quota group.