You can treat anything as a point mass as long as you're far away enough ;) And even with point masses only, orbital mechanics can get quite complex.
When you are close enough that a significant portion of the comet's mass is "besides" yourself (somewhere off to the side) it will pull you to the side.
This even holds true for spherical objects (say, you standing on earth), where you get gravitational pull not only towards the center but also towards the sides. For a perfectly symmetrical sphere, these sideway forces cancel each other out tough.
But earth isn't a perfect sphere, so even on our own planet you get (very small) variations in gravitational pull (stronger pull at the poles or near mountains (where the crust is thicker)): https://en.wikipedia.org/wiki/Gravity_of_Earth