Walking up a flight of stairs briskly is way above VO2 max for the vast majority of people, yet doesn't feel "extreme"
..In the states. Ask people from Amsterdam or Berlin about running out of breath from stairs.
I even remember seeing a study that claimed that people who lived in higher floors with no elevator had in average better cardiovascular health, an easy peasy way to nudge people into the right direction.
But I totally agree on the second point, starting in the the 4th floor was a bit hard in the beginning for me too