Conduit is lightweight. I'm running it on a tiny VPS. Its main problem at the moment is unreliable Android notifications.
Also, how's your database RAM/disk usage? From what i understand conduit.rs is very optimized but still relies on a huge DB for matrix to operate its magic.