You should write an operator for inner checks.
This might be helpful:
https://learnk8s.io/troubleshooting-deployments
I am planning to do the same for a platform that I am building, and is deployed on prem. Let me know if this is an open source project.