In fact you should be able to.
> user-scalable
> Controls whether zoom in and zoom out actions are allowed on the page. Valid values: 0, 1, yes, or no. Default: 1, which is the same as yes. Setting the value to 0, which is the same as no, is against Web Content Accessibility Guidelines (WCAG).
https://www.w3.org/WAI/standards-guidelines/act/rules/b4f0c3...
From a practical perspective, what makes this bad is the screen reader on a mobile device reading aria-labels for items that aren't inside the viewport, not to mention the fact that it's too easy to zoom unintentially either by browser configuration or accidental screen touches. Contrary to popular "wisdom", web browser zoom is one of the most anti-accessibility features out there, because there's no good way to detect page zoom from the Web API implementations of the most common browsers. What is more accessible is zoom on the entire screen buffer, not the web page. Both android and iOS support this (with focus change support) as well as many desktop environments.