The GC's marking phase is very slow because it has to traverse the whole object graph each time, meaning you want to make the object graph as simple as possible. Arrays of structs are very efficient for the GC to scan (it can either collect the whole array or not), but working with structs is a nightmare because there's no polymorphism.
For Fluffy[1] we just used classes and tried to optimize the hotspots as an afterthought. I'm not quite satisfied with how it turned out, there's still a periodical 20-40ms lag in the game.
There is a great article about keeping heap complexity low: http://blogs.msdn.com/b/shawnhar/archive/2007/07/02/twin-pat...
[1] http://marketplace.xbox.com/en-US/Product/Fluffy-Operation-O...