Nobody is forcing you to use a library (much less a full framework) if you don't want to. Yes, they're often a pain, and often result in a shitty DX.

It's more that the big name libs for this use case have a lot of optimizations, both server-side (like raster and vector tiling, CDNs) and client-side (asset and viewport virtualizations, prefetches, offscreen canvas rendering, etc.) learned over decades of doing web mapping & graphics that don't seem to be implemented in your app yet. They're all open source, so you can look at their techniques if you want. Especially their performance stress test examples.

I don't mean to make this a philosophical debate about libs vs writing your own code. You asked for opinions about the performance of your app, and I'm just providing one person's feedback. It's better than anything I could've written on my own, but not as fast as the major libs.