Would it be fair to say that the biggest challenge with the buy button is the lack of last mile fulfillment (like shipping, returns etc) from Google/FB/Pinterest. Amazon on the other does a job at this
The buy button primarily fixes the UX friction of having to leave the experience, go through a horrible mobile web checkout flow, enter your information again, etc...
Imagine those replaced with a streamlined flow (ala amazon single click buy buttons)
I can't imagine shopping for physical goods from Google. From what I hear, their customer service is pretty bad. I think overall that this is a risky proposition for these companies, in that if they mess something up they will have a lot of bad publicity.