This is misleading.
We raise very few animals purely through grazing on non-arable land. If nothing else, they need feeding through part of the year in many climates, and more typically, supplementary feed to increase intensity. Pure free range non-arable grazing ruminants constitute a relatively low percentage of total meat produced.
A somewhat more common thing is to raise cattle on arable fallow land between crop cycles. This is better than keeping permanent pastures, but nowadays we still often supplement the feed or intensively finish.
Battery farms, which is purely fed on crops grown where humans could grow food, accounts for about 70% of beef and 99% of pork and chicken. Often the soy being grown in the rainforest is to support these animals.
The other thing is a lot of "non-arable" and fallow land is actually a lot more arable with modern agriculture than it used to be, it's just more profitable to grow animals. A lot of the plant matter that used to be considered as only usable for animal feed can also now be processed for further use in human food.
Meat production isn't quite as bad as people sometimes suggest, but it's still pretty bad. There are some cases where it is still a good option (e.g. low intensity lamb grazing on rocky/hilly terrain) and if we scaled our meat production to only these cases, we'd be in a much better situation, but there's also be way less meat.