There are some Google people on Twitter responding to concerns such as yours. @JohnMu is one of them. You can send a private message to him about it.
We've experienced similar issues, but possibly with a different cause, see https://seotool.ee/indexed-not-submitted-in-sitemap
Once Google doesn't like your site, it's not easy to get it back. It involves being patient and trying to understand what's wrong. Sometimes though, there is nothing to be done and your site just comes back up. Google is a big black box in this regard, and doesn't give any message in search console about what may be wrong.