It was never about "building infrastructure", though, which is why they used Compulsory Purchase to force farmers to sell their land for pennies. Because obviously "undeveloped" land without any sort of planning consent is worth very little.
Now those bits of land, which have been put through the planning system and can now be built on, are not being used for HS2. So, they're being sold back to the farmers, right?
No, they're being sold for thousands of times the purchase price to property developers run by the people who donate the most to the government.
It's a land grab, same as the "inheritance tax on farms" thing.