A casual, "Hey I'm going to lunch, care to join?" works well with my colleagues. Usually they decline, to which one can reply, "Ok, have fun. See you in an hour"
Work doesn't need to be about buddies. work is about work. That's why you have a personal life. keep the two separate.
corollary #1: dear manager. I am not your friend, you are not my friend. i will never be your friend. Chances are i don't like you at all. Just be professional, and i'll be fine with that.
I am curious: Are you going out for lunch alone? Or are you not going out for lunch at all? Or don't you have a lunch break?