The biggest GC issues I’ve personally seen manifest were arrays of historical data that grew to tens of millions of entries and due to array storage in .net, the array was placed in the large object heap. Swapping to a linked list actually fixed the issue and the team lived to fight another day.
Like a lot of premature optimization, it isn’t a problem until it is… but solutions aren’t unattainable.