Can you pin versions in apt? Last time I checked it involved a lot of work and swearing, but that might have improved lately.
What you're looking apt is apt-holding:
apt-mark hold libxfont1
That been around since 2013 IIRC. And there was dpkg way of doing it before.