If you mean GSM information it's not quite the same. You can also easily power most non-smartphones off, meaning they won't communicate with the tower.
You can theoretically turn off your iPhone easily as well. Unless your dumbphone has a removable battery it would be just as difficult to trust it's actually off than a modern phone. But sure, removable battery would be an advantage on that dimension, I agree.
It's a hassle but you can carry the turned-off phone in a small metal box. Feels bad but should work.
There was a story on HN a while ago about municipal workers putting their phones in potato chip bags so they couldn't be tracked. Apparently it worked.