The Pantheon is just an example, but many other instances can be found in Rome: the Church of Santa Maria degli Angeli, the Basilica of Saints Cosmas and Damian, etc. (These are just examples you find in Rome, there are plenty of other examples here in Italy.)
The OP suggested that the loss of architectural art is mainly due to a clash of different cultures and to religion. While there were cases where this was surely true (the many churches destroyed during French Revolution!), in most of the cases the reasons were much more practical and did not involve conflicting cultures or the «long tentacles of the Catholic Church».
Before the concept of «historical preservation» was invented, what should have been the point to keep a temple to Venus in the crowded center of a city, when nobody worship her any longer and that very space could have been used for something more useful?
These acts of destruction happen all the time even within the same cultural context. For instance, the old Vatican Basilica, built in the IV century, was destroyed and rebuilt in the XVI century by the Pope himself! And the same applies to countless of other important ancient churches and public buildings (My favourite example: the astonishing church of St. Clemente [1], well worth a visit if you're in Rome! The XI-century building was erected over the old VI-century church.) Not only churches, but frescoes in churches have been continuously covered by newer frescoes, once the taste of the church goers changed. A few days ago I visited the beautiful church of Sant'Abbondio in Como [2]: the church dates back to the XI century, but the old frescoes were covered by newer ones in the second half of the XIII century.
Even the Romans did the same for many of their buildings: once they realized that they were no longer useful or usable, they either repurposed them or destroyed and built something else on the rubble. The Colosseum was not built on pristine land! And Nero destroyed a lot of houses to build his Domus Aurea, which was later destroyed by Vespasian and Titus and covered by newer constructions, with the intent of make people forget Nero. (No, as far as I know neither Nero nor Vespasian or Titus were catholic.)
It's a pity for scholars, but before historical preservation was a thing, it was often the best thing to do, as the older buildings did no longer fit their purpose. These acts of destruction made the world lose important pieces of art, but they were meant as a way to repurpose the spaces of the city according to the evolved needs of its inhabitants. This is particularly true for cities like Rome, which have been populated continuously for ~2500 years.
[1] https://en.wikipedia.org/wiki/San_Clemente_al_Laterano
[2] https://it.wikipedia.org/wiki/Basilica_di_Sant%27Abbondio