Yeah, I think you are right. Apps don't need access to GPS, internet or bluetooth to know your location. They can just log what cell tower you are connected to, and use that information to "estimate" your location every time you switch cell towers.
I think that method even works in flight mode, as your phone is still listening for cell towers, but just not responding. But I think this info is limited to the OS, and can't be used by apps.