There is an AI app for this already:

https://chatfai.com/characters/book

I haven't tried it, but if I had to guess how it is built they're probably just setting up RAG vector databases at a per-book level and then augmenting a given character's context window with information from the vector database relevant to the conversation.

It would be relatively trivial (weekend project) to roll your own using streamlit + quant/pgvector + ggerganov llama.cpp and a suitable model such as Vicuna/Mistral/etc. Hardest part would be separating an entire book into a well representated set of embeddings.