Aren't you basically describing SSPL? That's MongoDB's license that does pretty much what you're describing, except it's not considered open source [1]. And in response, Amazon created an API-compatible alternative called DocumentDB [2]. There's also the Commons Clause [3], which I think Redis Labs uses to license its Redis satellite products.
[1] https://www.zdnet.com/article/mongodb-open-source-server-sid...