Or, it could be that they are not amazing at managing you. They are not helping you find solutions to the problems you are having, nor are they mentoring you to resolve the problems yourself. As a result you are diminishing your own presence and work. And they are ok with that?
To answer your question, I would not switch. It sounds like you mostly like the job, you just dislike a couple co-workers. That will always be the case, anywhere you go. Part of learning to navigate your career is learning how to work with people whose working and communication styles differ from your own.