Rolling your own auth system is very likely not going to produce an auth system as battle hardened/tested as <popular framework>.
Rewriting an entire site from scratch is rarely the correct choice and is one of the most common engineering fallacy. https://en.wikipedia.org/wiki/Second-system_effect
Caching is a problem you need to solve regardless of whether you're using a framework or not.
These problems don't magically go away when you decide to rewrite these components yourself. You probably aren't going to write code as well as a community of coders that have contributed to a years-old codebase. Especially not on your first pass.
These are all pretty legitimate "what-if" questions to ask.