My impression is that although it is not spherical, the core of the comet is a single object. It may therefore be treated as a point mass for these purposes.
When you are close enough that a significant portion of the comet's mass is "besides" yourself (somewhere off to the side) it will pull you to the side.
This even holds true for spherical objects (say, you standing on earth), where you get gravitational pull not only towards the center but also towards the sides. For a perfectly symmetrical sphere, these sideway forces cancel each other out tough.
But earth isn't a perfect sphere, so even on our own planet you get (very small) variations in gravitational pull (stronger pull at the poles or near mountains (where the crust is thicker)): https://en.wikipedia.org/wiki/Gravity_of_Earth
Thanks for the correction; sorry for the noise.