You could run it off of a laptop sitting in your closet and be able to support 3-5 simultaneous users no problem. It'd be a benefit if you had 1GBPS symmetric fiber, though.
But yeah, RPi is rough because you do need legit CPU to transcode on the fly, which is what Plex does, and the more users the more transcoding power you need.