I'm also curious what prevents say an amateur operator or even another country from sending commands or disrupting communication to remote systems on Mars. Is it the just the amount of work they'd have to do, proprietary/secret protocol, encryption?
The practical requirements of getting a strong enough signal to Mars would require a big dish to focus the energy and some high power microwave equipment which make it more difficult but not impossible given enough resources. Then as you mentioned one needs to know the communication protocols.
BTW there were Ham built satellites in orbit, that piggybacked onto other missions. (OSCAR) Not sure if they are still in orbit. And again they used reserved spectrum for licensed experimental Amateur radio stations.
For example hams can only transmit in the 1.25 meter band if they don't interfere with other users like the Automated Maritime Telecommunications Systems (AMTS).
The occasional amateur gear is usually not enough to demodulate signals from Mars, since they're too faint. You'd need much larger dishes to, roughly speaking, "ingest" more energy into your receiver so that it become more distinguishable from noise and you can extract information from it.
It would be possible to demodulate with smaller, amateur dishes if the spacecraft transmitted in a very narrow bandwidth. It wouldn't make sense for space agencies to do that, though, since they would trade higher bit rates for cheaper receivers, whilst powerful receivers aren't that difficult to build and are totally on budget for an agency capable of launching a spacecraft.
But I must say that that would be a very nice citizen science project.