Frozen fruits like berries are typically really budget friendly, as long as they get frozen shortly after harvesting they contain the same amount of anti-oxidants and taste fine in my experience, although might require a bit more planning.
It isn't that good reasonably priced vegetarian and vegan lifestyles aren't possible. The problem it's when you set "cheaper than animal products" as the bar to get under, it becomes very boring very quickly.
I do limit the amount of animal products I consume myself, and unless everything is in season somewhere somewhat local, I don't find myself saving a ton of money.