That simply isn't true. Your device does load a library (usually minified), and parses the JS to be sure it is runnable. But it doesn't run every function just because it is loaded in memory. Even if it did, it happens on app load, once... then runs functions on-demand. And the size of the load can be mitigated with appropriate use of tree-shakable libraries. But without JS, the server regenerates an entire page, every time you do something, which takes some resources. It sends it over the wires, which takes some small amount of resources. The mobile device loads an entire page again.
I do agree there is wasted processing in both scenarios. But that waste doesn't just disappear when you do server-side work.
We all can over approximate in our preferred direction.