When code is contributed to a FOSS project, and there's no CLA, the contribution is licensed by the third-party contributor under the same terms as the project, and furthermore the contributor retains copyright (assuming there's no CAA, which is stronger than a CLA).
So if you start an AGPL project and offer a paid SaaS, you need to do one of the following:
a) don't accept outside pull requests
b) accept outside pull requests with a CLA or CAA
c) provide the full source code to the paid SaaS to users, as per the terms of the AGPL
d) keep those pull requests in the AGPL repo but rewrite or remove them in the paid SaaS
Any other path opens you up to risk of copyright infringement lawsuits if your business is successful.
The OP eventually added a CLA, but had already accepted a lot of code contributions prior to that. So in their previous Show HN, I asked what path they took to resolve the AGPL hurdles, and they did not respond.