TL;DR: algebra != arithmetic AKA real math doesn't use numbers

I'm only good at _arithmetic_ because of making Warhammer 40k armies (true story bro).

I'm good at _algebra_ because I was taught well, on top of a knack.

Speed with basic algebraic operations was very helpful in many places but speed with arithmetic operations has only been helpful in board games.

I don't think anyone here would disagree with your point about algebra, but I think a lot of people such as myself would disagree that pre-K memorization of arithmetic helps with algebra later.