Let's say it's 100°F, and you're outdoors for 2 hours. You feel lethargic and sapped of energy, and you look forward to going inside. When you finally do, you get a gigantic cold beverage, take off your shoes, plop down on the couch, and wait to feel normal again. It might take 30 minutes before you start to get there.
If it's 65°F inside, this process happens faster than if it's 75°F.
No doubt, the moment you get inside, you already feel better whether it's 65°F or 75°F because neither one is 100°F. But if you're alternating between inside and outside a lot, colder is still preferable.