Yeah, this is a stupid thing that stems from a sound shift in Ancient Greek about 2000 years ago, when Φ changed its pronunciation from [pʰ] to something like [f]. We have all kinds of evidence of more or less when and where this happened, but not why, although it's a thing that's happened in other languages too, where a phonetic distinction that's originally aspirated/unaspirated becomes a fricative/plosive distinction.

English has this stupid policy of keeping the original spelling of any loanword it picks up, gradually drifting closer and closer to being an ideographic script. In this case English got its Greek morphemes by way of Latin, where they'd mostly already acquired standard spellings based on the older pronunciation. In Spanish, which has a phonetic script, it's much saner: "panel fotovoltáico".

Maybe we should start calling them ΦV panels.