Not quite true. If you are driving (say, along I-95 on a long trip) it's true that your phone will disassociate with one tower going out of range to a new tower coming into range, and the phone switches behind the towers will transfer "ownership" (more like association), but once you (if you aren't driving, or your passenger) accept a phone call, things get a bit different.
The cell phone is answered. So, it goes from switch O (the caller) to switch A (where your cell phone is associated with). Eventually, you'll move out of range of towers for switch A, and come into range of towers for switch B. But because the phone network is circuit switched, you can't just create a circuit from O to B. No, what happens is that A forms a connection to B, so now the call is going O -> A -> B. Talk long enough, and eventually, your call may end up going O -> A -> B -> C -> D -> E -> F.
So while it's true that your call doesn't go through multiple towers, it does go through multiple switches (as long as you are talking on the cell phone, and the cell phone is moving).