> going from MIT to GPL is fine, but GPL to MIT is not
AFAIK both are _not_ fine, eapecially because the licenses are totally incompatible.
When you publish a patch under a certain license (that of the project at the moment of the contribution), you _only_ allow using your code under that license. This is why dual licensing is a thing.
Except when you sign a CLA - mostly used by companies to gain copyright over your work. In that case the company can do whatever they want - even charge you later to use your own code.