So you're actually right more than you think. Plenty of teams that make money do stop making money over time (or make less). I'm sure there were teams at Jane Street over the years that have folded or dissolved because they stopped making money. But as a firm, they may be making more money.
In the HFT firm I used to work for, that one team went from making $500k/day to $0/day. But another team went from making $30k/day to $2M/day. That's right, a team of 12 people making $2M/day. They probably ate most of the former team's lunch. That's how it goes, survival of the smartest and fastest.
So why hasn't Jane Street's edge been arbitraged away? Some of it has, probably by people within Jane Street who found even more edge. When you have hundreds of the brightest minds in the world, who's going to eat your lunch but yourself (and a few other high quality trading firms)?
But ultimately your assumption is wrong. What they do is not simple at all. It is not like building a simple CRUD application. Throwing money at the problem does not mean you will succeed. You will most likely fail. Strategies that used to work may no longer work when markets change, and markets are always changing. So some traders may try to take their ideas and apply them elsewhere, and they may have some limited success (not to mention the high quality speed and infrastructure you need to win), but when the market changes in 6 months, are you smart enough to keep up?