Most masterpieces where literally hanging in the background of some rich person's summer houses, and hunting lodges, and other properties. Thrown out and replaced on a whim.
Art was used by the rich and famous to show off their wealth.
In the 19th century countries got into the game (the idea that Dutch masters were ending up in American collections was a national embarrassment).