If it were a coincidence, I think this would be fine. But if you ask someone whether they want to meet up, and they say no, maybe just leave them alone.