You don't need (and shouldn't depend on) packagist when using Composer. You can import packages directly from Github, Bitbucket, etc., or even locally, you just have to write a bit more JSON.
But it is possible to avoid packagist. Ugly and painful maybe but possible, and YMMV regarding the effort being worth the reward. Dependency management in PHP isn't (or shouldn't be) like it is with JS/Node where there is one authoritative registry that everyone has to depend on and that can assert direct influence over the community as a result. Packagist shouldn't be the PHP package registry, but a registry.