The author talked about its effect on compile-times and I have to say, I agree with him. It's also why I dislike header-only libraries, they also bloat the compile times unnecessarily.
Don't get me wrong, the fmt library is very nice, but you can't deny its effect on compile times.