The 20 bucks a month is actually for 2TB of storage. Gemini advanced is just a plus.
Edit: I found the answer on Reddit, as usual. You need to go to your plan settings and only then is the plan available under one.google.com/settings. That's the only way to downgrade.