Once a codebase reaches a certain size and level of complexity—and is clearly no longer a toy project—it becomes difficult, and eventually impossible, to keep every implementation detail in your head. At that point, this also starts to drift away from what you should actually care about: invariants, and ensuring they are not broken or gradually eroded over time. From my point of view, this is much harder to achieve than one might expect, especially without a highly skilled team to help reason about and discuss these concerns.

Focusing on concepts and invariants, and being able to reconstruct the core ideas of the system from them, is therefore the right priority.

Keep grinding, and good luck with your project.