If it's a GC thing, you should also try using various D compilers to compare performance.

There is:

dmd (official D compiler),

ldc (llvm D compiler),

gdc (GNU D compiler).

When I was messing around with D about a year ago, dmd compiled code the fastest, gdc by far the slowest, but gdc produced the most performant code. ldc was middle ground for both compile speed and program performance. Perhaps the GC issue you are having is just a quirk of the current compiler you are using.