In the current model every plugin you'd have in your repo would have to be aware of your plugin manager, right?
I suspect you'd need adoption before authors would want to maintain a Pluginfile.json, and before there's a decent selection in your repo there's not a selling point to gain adoption.
A tool that can help me find things like git-series[0] would be cool, but I think you'd need to work downstream rather than having every git plugin know about your package manager. If you're gonna make repos to wrap upstream projects into your package format (which is what Linux distros do), then that could work.