You don't owe random strangers your time, and it's so strange to me that people feel so entitled to other peoples time. Can also be argued that it's rude to engage a person on a topic they're not interested in.
You could argue, in your description, the same about RMS - he might feel entitled to someone's time to talk about free software.
RMS set boundaries and that's perfectly reasonable.