Well, no. That’s not true. Firstly there are rules about the altitude you can fly at when it’s greater than 3000’, and second you are required to file a flight plan any time you are flying IFR.
And why are you bring up IFR? Most private pilots are flying VFR, even if they are IFR certified. But even in controlled airspace you don't have to file a flight plan. I can take off from a class C, fly through uncontrolled space and am not required to talk to anyone again. I don't have to say anything until I decide to land.
This is pretty common among pilots too. I'm not sure why drone pilots should be held to higher standard. Really they should be held to lower.
Drone pilots have to be held to a higher standard, because they have no skin in the game.
If a plane pilot flies like an idiot, they might hurt other people in the air... But they are very likely to also get themselves killed in the process.
This creates an incredibly strong incentive to not fly like an idiot.
A drone operator flying like an idiot is far more likely to get other people killed, than they are to get themselves killed.
If drone pilots could only operate them, while wearing a suicide vest that explodes if the drone crashes, I'd be more open to holding them to a lower standard than pilots.
IF something catastrophic happens. Issue is that a 2lb drone is much less likely to do damage than a plane. You seriously can not compare the two. The danger levels of the two are nothing alike. If they were, I'd agree with you, but these things have magnitudes of difference in danger.
If you want to get pedantic, you're also required to file a flight plan if there is some restriction in place requiring such for the area you are flying in. For example a TFR, or operation in an ADIZ or SFRA/FRZ(Washington DC). Or if you are flying internationally.