The closest mention I can find is this [1]:
> Early monastic rules say little about how time was determined, though the Rule of the Master notes that pairs of monks took turns, in weekly shifts, trying to stay awake while the others slept in order to awaken the abbot on time. Several possibilities for deter- mining that time are likely, however, including the use of calibrated candles or lamps of oil, [...] the recitation of a certain number of psalms, [etc]
However, here is it only mentioned as a possibility, not an actuality, and no sources are provided.
There appear to have been several orders who practiced 24-hour prayer by taking shifts, see for example the Acoemetae [2]. Here we do have Monks staying up all night reciting psalms but I can't find any indication that the prayers themselves were used to keep track of time.
The only reference to prayer-as-timekeeping that I can find seems to be in Medieval recipe books [3], which apparently would say things like boil this fish for as long as it takes to recite the lords prayer three times. Dead-reckoning works well on such short time-scales!
Even as early as the 500s there appear to have been timekeeping methods accurate enough to tell that dawn was approaching. When the skies are clear you can read this from the stars and accurate-enough water clocks have been in Europe since ~ the 500s. So there doesn't seem to be a need to perfect the act of spending hours reciting psalms at the right tempo.
[1]: https://libres.uncg.edu/ir/uncg/f/M_Helms_Before_2004.pdf [2]: https://en.wikipedia.org/wiki/Acoemetae [3]: http://the-history-girls.blogspot.com/2017/08/cooking-on-pra...
If I had to pick a "best-of" reading list, I'd say the 9th-century "Musica Enchiriadis", Berno's "Prologus in Tonarum", and Boethius's "De Institutione Musica". These all assume a familiarity with Platonic music theory, so at least skim through Plato's "Timaeus" first. There will also be more references to astronomy than will make sense to somebody in the modern age, because music and astronomy were believed to be essentially the same thing. Also more recent research has found that actual musicians of the time seem to have completely ignored the theorists. Some things never change.
In the 18th Century Presbyterian Church, they only sang Biblical psalms set to simple (but distinct) tunes and without any musical accompaniment. This was known as exclusive psalmody.
The psalms weren't the prose-style you'll find in any print Bible though, they were their own translation called the Metrical Psalms. Its 1650 edition proved very long lasting.
Many of these psalms were set to common metre, meaning that if you found such a psalter and tried singing it to some well-known tune it will very likely 'fit' the lyrics. You could try Psalm 23 set to the tune of The British Grenadiers, or Amazing Grace, or even The Star-Spangled Banner: https://thewestminsterstandard.org/psalm-23/
This was actually how the psalms were practised by congregations in my home country. The psalm lyrics were considered too sacred for practice, so they set the same tunes to other (sometimes comical) lyrics and practised those during the week. Then on Sunday the minister would say something like 'we will now sing Psalm xx to the tune of lilies', and the congregation would know the tune even if they hadn't memorised the psalm lyrics.