I think a differentiation between a full framework (like Next.js) and a lib (like a Canvas rendering lib) is helpful here. There's a lot more that goes into modern web graphics rendering than just caching, including optimizations within JS itself, its interactions with Canvas and the DOM, WebGL, WebGPU/WASM for expensive calculations, virtualizations, tiling, transformations/filters, etc.
I'm not a graphics programmer so I can't get into much detail here, but I think of it as the difference between making a game in an existing engine (Godot, Unity, Unreal) vs trying to make your own engine and rendering pipelines. It all compiles down to machine code, but their optimizations can be several orders of magnitude better than what a solo dev can realistically write in their own time.
With Canvas rendering, there's been a lot of techniques that companies have worked on over the years. Take a look at Felt, for example, to see a full web app that makes extensive use of render optimizations. They have a nice writeup about it: https://felt.com/blog/from-svg-to-canvas-part-1-making-felt-.... Figma too, but it's less a direct comparison to your app.