> Note the use of pointers, this forces other systems to take care of the actual storing of objects, leaving the physics engine to worry about physics, not memory allocation.

Based on the access patterns in the code (iterating through every physics object every step), keeping the physics objects behind a pointer is actively cache hostile! If at all possible try to have these objects in contiguous memory.