The people fed up with the non-conservative Tories weren't the ones who elected Labour. They stayed at home and didn't vote for anyone so the Tory vote collapsed, allowing Labour in.
Remember, Starmer won with three million fewer votes than Jeremy Corbyn got! And nobody claimed Corbyn wasn't far left. It was called the "loveless landslide" because so many voters just turned away in disgust at what British politics had become, allowing Labour to win by default rather than because they were popular.
Partly this happened because many of the supposedly Conservative MPs preferred a Labour government to giving their own voters what they wanted. Sunak called a surprise snap election to deliberately ensure Farage (the only competitor on the right) would have almost no time to prepare.