Exactly. My wife tutors kids at the high school who never mastered arithmetic and are trying to learn algebra. It's hopeless.
Fluency in arithmetic isn't something drilled into kids just to be obnoxious, it's foundational to almost all future math skills.