We are not pitching this as "THE" solution, but as "a" solution that can scale up to and beyond the global level of yearly CO2 emissions. Even after the "low-hanging opportunities" it makes a good amount of sense, as mining olivine is not difficult and there are large reserves on every continent.
You are incorrect that olivine mining at a large scale would incur tremendous energy costs, it will not. I would suggest you check out this model of a 5,000 tonne per day open pit mining for porphyritic rock. (This model also includes 5,000 tonnes per day of "waste" rock which will likely not be wasted in our use case) http://costs.infomine.com/costdatacenter/miningcostmodel.asp...
In this model, which is not in any way optimized for environmental efficiency, it requires a diesel fuel quantity of around 4,751 liters/day to mine the 5,000 tonnes. At 2.68 kg of CO2 per 1 liter of diesel, that generates 12,732.68 kg of CO2 per day. That 12,732.68 kg = 12.73268 tonnes of CO2. The 5,000 tonnes of olivine mined will eventually weather and sequester 6,250 tonnes of CO2 (1 tonne of olivine sequesters 1.25 tonnes of CO2), for a net capture of 6,237 tons at the mine.
We have life cycle assessment that calculates the CO2 penalty and loss on efficiency, including milling and transport to locations less than 300 km (186 mi) at around a 4% loss [1].
Just for your information and for others, from a financial perspective the mining in that model costs $7.32 per ton, and then the transport and milling/crushing only costs about $3 per ton, so olivine could be transported to a beach at around $10/ton, with the price per ton of CO2 sequestered at less than $10.
You are also incorrect regarding the mining impact. The mining of things like shale requires fracking and the injection of sand and chemicals, our mining is simply open pit. Essentially, you open a pit on the surface and simply dig it up. As mentioned above, it is not all that energy intensive either.
For global CO2 level removal in terms of mining, it would likely require 30-50 mines in the wet tropics, preferably at a level of greater than 100 million tons/year (due to economies of scale).[2] There are large reserves on every continent and plenty near coastlines[3]. If you wanted to open fewer mines, you could theoretically find a few large reserves. For example, there is an open pit mine in Bingham Canyon that has an excavated volume of over 25 km^3, which would be the equivalent of 2-3 years worth of the volume of material needed.
Don't worry about this idea getting "tossed out with the bathwater," if anything we will actually be removing CO2 from the bathwater and de-acidifying it at the same time ;)
[1] Environmental Life Cycle Assement of CO2 Sequestration Through Enhanced Weathering of Olivine https://projectvesta.org/science/#dflip-df_978/1/
[2] https://projectvesta.org/science/#dflip-df_77/9/
[3] https://projectvesta.org/science/#dflip-df_90/25/