I don’t know where you are getting data for ruminants in North America. Historical estimates for bison are 60 million prior to systematic extermination, at least 10 million for elk, 50-100 million for deer, plus pronghorn, moose, etc in the many millions each.

Bison may not rebound very quickly but something needs to fill the bison niche in the ecosystem. It is well-understood that several other native species undergo population declines and reducing biodiversity when beef herds are moved out of a region long-term. Cattle and bison are very substitutable for this purpose (and are related species). Deer fill in much faster than most other species of ruminants in North America, usually detrimentally.