There are a few things that make moving your calendar away from Google pretty hard. If a Google account invites you, the invite still lands in your old calendar for example - even if that's not where your domains MX records point. Lots of third-party software also has mostly Google-specific integrations.
Moving away from them is hard, but overall probably worth it, iff you move to a provider that has real support.