In math everything is persistent/immutable anyway, so persistent data structures are a given when working with purely mathematical abstractions.
There is a memory issue because of this, but this problem isn't exclusive to FP languages. Many languages solve this with something called garbage collection.