No sense using exFAT because it's not as widely-supported. Don't have to reformat fat32 because File Allocation Tables are extremely simple. Move all files off, and then move them back serially, and presto, no more fragmentation.
Except it's not by all of the things I need to use, so that's not going to work. Compatibility vs. 4 GiB limit. There is no perfect, only trade-offs.