Just keep your objects live, attach reusable scratch space to them, and memory pool them instead of throwing them away.

(from my experience anyway in writing critical-path allocation-free C# with Unity).