Can't say it'll solve all your problems, but try Florence-2. It's worked well on some handwritten documents for me when all the text is a relatively uniform size.

Model: https://huggingface.co/microsoft/Florence-2-large

Demo space: https://huggingface.co/spaces/gokaygokay/Florence-2