What I tend to do is have a dropdown with a country icon and language name localized to that language for each language. My pet peeve is trying to switch a website from a language I don't speak into one I do speak and the dropdown listing "Dutch" as "Néerlandais" or whatever instead of "Nederlands". I don't speak french, why would I know what dutch is called in french?
(and yes, language switching menus you can't understand are dumb, especially for more complicated cases where names don't even look similar, but unfortunately common)
Linguists try to draw some lines in the sand, but in the end the reasons we call Swiss German a dialect and Luxemburgish a language are cultural and political.
As opposed to a country, which needs an airline and a beer: https://quoteinvestigator.com/2014/09/16/beer-airline/
As a native German speaker, who never learnt Dutch, I can tell you that I understand nl-NL a lot better than de-CH. There is a reason de-CH is subtiteled in German media.
I do think it might be conceptually possible and intellectually funny to come with flags that are generated from some pattern that some corpus of languages exhibit. Be it sui generis through some LLM, or relying more on scholar lexical/syntactic categorizations, or even something really more trivial like using a language-code[1] as entry.
On the other hand, with such an approach I’m afraid that chances are low that we can gain any adherence and a large recognition of the resulting flags from their speakers.
To my mind the best approaches as of today where already exposed in this thread: something best effort to propose the most likely user recognizable language and a clearly visible option to open a locale selector. If iconography seems a must have in the UI, showing something like a A/あ icon is rather usual by now.
You need just one icon to open the locale selector, and possibly an other one giving feedback of which one is currently applied. For the latter using a language code looks like a realistic option:
EN
US
FR
BE