I'm on relatively modest hardware and just scrolled my hardest using a trackpad in urxvt, xterm, xfce4-terminal, gnome-terminal, and they're all practically instant. You'd need to _introduce_ lag to have any pixel-by-pixel smoothing even be noticable.
I consider pixel-by-pixel scrolling unnecessary eye-candy not worth the tradeoff. Consider that the capabilities in both hardware and software to do smooth/pixel-by-pixel scrolling in terminals have been around for decades - the reason it's not more common isn't necessarily technical.
You may have a different preference but that doesn't make the status quo sucky, laggy, or sad.
Trackpad scrolling works fine for me, in everything I can think of. (macOS+Terminal.app => seems fine. macOS+iTerm2 => seems fine. Linux+Terminator => seems fine.)
(Terminator will even scroll to partial lines with the trackpad. It won't do the sort of scrolling described in the OP, though … but that's not "sucky laggy scroll", either.)
I wouldn't like smooth scrolling when I use PageUp/PageDown, unless it added minimal latency, because for full pages it would break the illusion.
However, it may feel natural and normal with small and precise movements like multiple finger trackpad scroll for a few lines - at least if there's an acceleration curve to reduce the fps for large movements (many lines, or quick movements to scroll by full pages): it would help keep the perception of a low latency between the action and its "immediate" execution.