I think there's a distinction between accruing and applying generic skills in a particular discipline and using intellectual property from one employer to benefit another. If you have agreed to protect the trade secrets of your (former) employer, you need to exercise your professional judgement to draw the line between contributions that stem from your experience in the field and those that are informed by work that you or others did for hire at your former employer.

Within that framework, plenty of things are clearly out of bounds, like copied source code. I would argue that re-implementing chunks of code that you know to be economically valuable and unique to your former employer from memory is just as problematic.

Now, by out of bounds, I mean breach of a civil contract. I don't see the criminal aspect.

Basically, there's no need to erase your memory, you just need to distinguish exchanging your time for money from exchanging your former employer's IP for money.