The trouble comes when you have data structures that manage memory, and you want to associate the allocation strategy with the particular instance of the data structure. Or, worse, you don't store the allocator pointer with the data and you require that all functions operating on the data structure happen to use compatible allocators.
At its simplest, though, passing allocators around is just removing an implicit global variable that is sometimes helpful to customize locally, but nearly impossible to customize locally if left as a global.