Also, when an OpenAPI spec gets sufficiently big, you face a need-in-the-haystack problem https://arxiv.org/abs/2407.01437.
Plan on doing a two layered system mentioned earlier, where the first layer of tool calls is as slim as they can be, then a second layer for more in depth tool documentation.
And/or chunking tools and creating embeddings and also using RAG.
It took us a decently long time to get the search quality good. Just a heads up in case you want to implement this yourself
First layer just describes on high level, the tools available and what they do, and make the model pick or route the request (via system prompt, or small model).
Second layer implements the actual function calling or OpenAPI, which then would give the model more details on the params and structures of the request.
Since 2 steps are required anyways, might as well use a dedicated semantic search for tools like in agents.json.