I've never experienced performance issues due to users. I run https://styleci.io and we're absolutely fine dealing with thousands of users.

If you're running jobs, be sure to do so on the queue.