I have been doing open source for 4-5 months now (main project has 1.8k stars on Github so far), so I am a newbie. However, I have always found ignoring people to be difficult. I immediately get a sense of guilt when doing it.
Keep it up and good luck. The most you invest in community, the more you can delegate and find "co-developers" who can help you in ways you couldn't imagine.