Commercial traffic flying at 30-40 thousand feet is not flying VFR typically. And of course spotting landmarks through cloud cover or at night is not a thing. But you are right that spoofing would largely just be annoying. Commercial traffic is usually in contact with a controller and monitored on radar. So, any course deviations would result in controllers asking what they are doing and be followed by them simply vectoring them back on course with compass headings.
However, GPS has replaced a lot of the NDB and VOR radios that pilots used to rely on. Also a lot of airport approaches are now GPS rather than ILS/VOR. It's pretty critical to modern aviation.
GPS, or rather GNSS (which includes GPS and other systems), spoofing is a concern; but so far mainly in conflict areas. Most airlines would route around those.
Resilience against spoofing is actually a big topic from a strategic point of view. Also, spoofing is a big part of counter measures against drones in e.g. Ukraine. So there is a bit of an arms race going there.
The point with GNSS is that most phones can utilize multiple satellite positioning systems at the same time at this point. Which is of course something that navigation systems in planes could also do. And of course many pilots actually use ipads in the cockpit, which they could use as a backup in case of an electrical failure in the plane.