This is a wrong way to think about it. You can say similar things about building a bomb. The steps to manufacture explosives from common materials are simple if you are good at chemistry. Likewise, this mathematical explanation makes encryption sound like a high-school can do it using an abacus. But it's not. It's simple for a person to understand. But coding these algorithms is no joke.
Now if you want the government to not impose restrictions on crypto software, that is the equivalent of wanting to lift controls off explosive substances. Yes these are fundamental truths about the world (mix a and b and get an explosion, multiply prime numbers and you can't factorize), but that does not mean the government cannot try to control them. Just like they control the sale of explosives, for public security, they can also require Microsoft, Apple and Google to put in a master-key. And I don't see a problem with that.