The practical requirements of getting a strong enough signal to Mars would require a big dish to focus the energy and some high power microwave equipment which make it more difficult but not impossible given enough resources. Then as you mentioned one needs to know the communication protocols.
BTW there were Ham built satellites in orbit, that piggybacked onto other missions. (OSCAR) Not sure if they are still in orbit. And again they used reserved spectrum for licensed experimental Amateur radio stations.