This behavior not only seems sensible to me but appropriate. If one uses pixels then one wants a pixel-perfect layout so therefore doesn't want fonts to scale.
CognitiveLens is right. Proportions matter.
(Note: when I say "you", I refer to the "one [who] wants a pixel-perfect layout" which may or not actually be you.)
So, if this is the case, then "px" units and raster graphics should not belong on the Web, too. But unfortunately, they do.
IMHO, the best approach would be to have two zoom modes. Some users prefer scaling text, but keeping images inact (so they won't see upscale ugliness), others prefer "screen magnifier"-like behavior. And none of those behaviors is wrong.