I once wrote a similar thing for building a custom LJSPEECH dataset out of ebook/audiobook combinations to synthesize my favorite narrator voices using coqui-tts and the VITS model and make them "publish" books that never came out as audiobook.
It was able to synchronize the book contents to timestamps, split the spoken word in to sentences and create a LJSPEECH datasets out of the combinations. I used aeneas[1], it was a bit finicky to set up, but after a while it even was able to map non-english languages (in my case german) with more than 80% accuracy. Worked out pretty well, the LJSPEECH datasets were good (I still have them here), but the TTS tech was not there yet :-) Maybe it's time to revive this project using newer modelling approaches like XTTS or something...