No, its not absurd.
Sure, I agree, for many things in life you can "learn by doing".
But this is cryptography.
There is no escaping that cryptography IS mathematics and an algorithm built on top of that mathematics.
Unfortunately the only way to learn the theory is by reading and understanding books or academic papers.
Otherwise you are just taking shortcuts based on a summary that someone else has written for you on a short blog or forum post. Which, aside from the fact that you are skipping the detail and hence not really learning, also brings us into the region of trust...
If you have no interest in learning the dry theory behind the cryptography, then you have no business rolling your own crypto. Because if you are not willing to understand the theory then you are going to make serious mistakes.