I fry most things in either extra virgin olive oil or butter. The olive oil stands up to the heat better than butter (the milk solids will burn).
I've been pan frying eggs, onions, pork chops, chicken, potatoes, fish, basically anything interchangeably in extra virgin olive oil and butter/ghee for decades. There is certainly a flavor difference, but that's because the olive oil is much milder in flavor than butter, and the process of frying dulls the flavor of the olive oil, as mentioned.
I - along with millions of people in countries bordering the Mediterranean - enjoy the flavor of food cooked in olive oil.
Adam Ragusea did a great video on the topic, with cited sources. I highly recommend watching it:
https://www.youtube.com/watch?v=l_aFHrzSBrM
Sources:
https://actascientific.com/ASNH/pdf/ASNH-02-0083.pdf
https://www.sciencedirect.com/science/article/abs/pii/S02786...
https://www.mdpi.com/1420-3049/24/8/1555/htm
Including this famous chef using it for deep frying:
https://www.youtube.com/watch?v=5ZSPDYAUn7E
Additional sources:
https://www.nutritionadvance.com/cooking-with-olive-oil-good...
https://www.bonappetit.com/story/can-you-fry-with-olive-oil
https://pubmed.ncbi.nlm.nih.gov/24360472/
https://www.healthline.com/nutrition/healthiest-oil-for-deep...
The bottom line is that extra virgin olive oil is more heat stable than other cooking oils with higher smoke points. Just don't use it to sear steaks or something.
So much of the conventional wisdom around cooking is not based on scientific fact, but rather on unsubstantiated observations and gut feelings. The viability of extra virgin olive oil for cooking is one of those pieces of faulty conventional wisdom, the need to sear steaks to "seal in the juices" used to be another big one, until we finally got the record set straight.
It's an extremely interesting subject of research, which may hold a large part of the explanation for the apparent health benefits of the Mediterranean diet :-)