How about a multi-headed approach. LDMs specializing in certain data operating under one LLM that acts as our point of access?