lbeurerkellner

there is an LVE for this: https://lve-project.org/reliability/repetition/openai--gpt-3...

This works until it doesn’t: https://lve-project.org/blog/how-effective-are-llm-safety-fi...

LMQL author here. You may want to try the preview 0.7b release of LMQL, chat is not in the stable 0.0.6.* series. Lots of things are coming in LMQL also, so don't worry :)

Thank you for bringing up LMQL. We have active branches with regex, parsers and types (structured and other types) which will also soon be upstreamed, improving typed LLM use beyond the current template-based approach we support.

LLM calls are just function calls, so most functional composition is already afforded by any general-purpose language out there. If you need fancy stuff, use something like Python‘s functools.

Working on https://github.com/eth-sri/lmql (shameless plug, sorry), we have always found that compositional abstractions on top of LMQL are mostly there already, once you internalize prompts being functions.

I think one should not underestimate the impact on downstream performance the output format can have. From a modelling perspective it is unclear whether asking/fine-tuning the model to generate JSON (or YAML) output is really lossless with respect to the raw reasoning powers of the model (e.g. it may perform worse on tasks when asked/trained to always respond in JSON).

I am sure they ran tests on this internally, but I wonder what the concrete effects are, especially comparing different output formats like JSON, YAML, different function calling conventions and/or forms of tool discovery.

It's interesting to think about this form of computation (LLM + function call) in terms of circuitry. It is still unclear to me however, if the sequential form of reasoning imposed by a sequence of chat messages is the right model here. LLM decoding and also more high-level "reasoning algorithms" like tree of thought are not that linear.

Ever since we started working on LMQL, the overarching vision all along was to get to a form of language model programming, where LLM calls are just the smallest primitive of the "text computer" you are running on. It will be interesting to see what kind of patterns emerge, now that the smallest primitive becomes more robust and reliable, at least in terms of the interface.

If anything this removes a major roadblock for libraries/languages that want to employ LLM calls as a primitive, no? Although, I fear the vendor lock-in intensifies here, also given how restrictive and specific the Chat API.

Either way, as part of the LMQL team, I am actually pretty excited about this, also with respect to what we want to build going forward. This makes language model programming much easier.

LMQL dev here. A bit of an unusual way of getting highlighted on here, I must say.

I am sorry to hear that you have issues installing LMQL. We are working very hard on LMQL, to make it as seamless and accessible as possible (as you have seen with our docs). However, we are of course not a well-funded company and the project is open source and community-driven.

Please feel free to reach out (hello@lmql.ai or Discord), so we can assist you in getting started. Please also consider contributing to the project, to improve it with respect to the kind of issues you were experiencing.

It's still a bit hacky in the current PyPi version of LMQL, but you can also use it from the command line, just like `python -c`:

  echo "Who are you?" | lmql run "argmax '\"Q:{await input()} A:[RESULT]';print(RESULT) from 'chatgpt'" --no-realtime

Gives you: I am an AI language model created by OpenAI.

I am one of the LMQL devs and we plan to also add a little more seamless CLI interface, e.g. to support processing multiple lines of text (e.g. quick classification tasks).

Yes, you are correct. LMQL has its own interpreter and runtime, that executes and translates your query programs into multiple LLM calls, required to satisfy your overall workload. It is inherently multi-part and automates several consecutive calls to the LLM, based on the query program and prompt template that you provide.

Re Node.js: We are actively investigating LMQL use from other languages than Python (see also https://github.com/eth-sri/lmql/issues/1). The current interpreter is quite closely tied to a Python environment, meaning to use it from Node.js, you will at least have to host a Python interpreter as a subprocess for now. We are actively looking into gRPC/inter-process communication though, hoping to improve on this a bit in the near future.

For a showcase of different LMQL queries and resulting model output, have a look at https://lmql.ai.

Per query, the LMQL runtime calls the underlying LM several times, to execute the complete specified (multi-part) query program.

It does not only translate to only one LM prompt, but rather a sequence of prompts, where during generation additional constraining of the LM is applied, to ensure the LM behaves according to the provided template. That's how it support control-flow and external function calls during generation, it actually executes the queries with a proper runtime and only uses LLMs on the backend.

If I understand correctly what you are imagining is some sort of local retrieval enhancement to generate a specific part of the overall response, which later is removed, once e.g. the generated piece of code (e.g. a function call) has completed generation.

Indeed, this is a form of LLM prompting that we are also exploring in our preview release channel with something called in-context functions. See https://next.lmql.ai and choose the "In-Context Functions" example in the New Features showcase screen.

With in-context functions, you can provide additional instructions/data to the LLM, that will apply locally only. Once such an in-context function returns, additional instructions/retrieved info is removed and only the LLM-generated end-result remains.

Author here, thanks for posting us.

Be sure to play around with our entirely web-based playground IDE here: https://lmql.ai/playground and our example showcase at https://lmql.ai. We are happy to answer any questions that come up.

Author here.

LMQL gives you a concise way to define multi-part prompts and enforce constraint on LLMs. For instance, you can make sure the model always adheres to a specific output format, where parsing of the output is automatically taken care of. Also abstracts a number of things like APIs and local models, tokenisation, optimisation and makes tool integration (e.g. tool function calls during LLM reasoning) much easier.

In practice this saves you a lot of ugly text concatenation and output parsing code, letting you focus on the core logic of your project. Overall, however, you will still use your host language to call LMQL. E.g. we are fully integrated with Python, where LMQL query code simply lives in decorated functions (https://docs.lmql.ai/en/latest/python/python.html).

We are building a programming language (https://lmql.ai), that allows you to execute programs with control-flow and constrained behavior on top of LLMs. Essentially imperative LLM text generation.

LMQL dev here :) Can you specify what you mean "constrains its utility". Are you looking to use LMQL from different languages? We are currently thinking about expanding to other ecosystems, so it would be interesting hear your thoughts.

I think prompting will definitely evolve more towards a PL than informal natural language, as informal text will always be inherently imprecise.

I don't fully buy into this idea. In the end there is an inherent bottleneck when it comes to the expressiveness of natural language. To do something correctly, you need some sort of precise description of what it is that you are trying to achieve. Unless future models can read you mind, there will be room for some sort of precise language to specify what you are trying to achieve, don't you think?

Definitely. I see this going even further, probably leading to something akin to programming, i.e. interleaving multiple prompts, one prompt calling another, like lmql or langchain.

(Another LMQL author here)

Cost is definitely a dimension we are considering (research has limited funding after all :) ), especially with the OpenAI API. Lock-step token-level control is difficult to implement with the very limited OpenAI API. As a solution to this, we implement speculative execution, allowing us to lazily validate constraints against the generated output, while still failing early if necessary. This means, we don't re-query the API for each token (very expensive), but rather can do it in segments of continuous token streams, and backtrack where necessary,

This is still more expensive than doing all in one request, but it is an inherent limitation of the OpenAI API, and not LMQL. On the upside, you gain more control, scripting and constraints, even with OpenAI models.

Ideally, some program representation of a scripted prompt like LMQL queries could be send over to the inference service, and be executed locally with full model access. This way, model vendors would not have to expose their models fully (e.g. to protect against distillation), but API users would gain a lot more control and efficiency. Alternatively, of course, better open source models with full access to logits are the ultimate solution, which is also the context in which LMQL was initially conceived in.

LMQL author here, thanks for commenting. prlang looks really cool, I will have a closer look. LLMs+PLs is a very interesting field right now, lots of directions to explore.

LMQL`s efficiency gains can be attributed to close supervision of the generation process, as the token masking via constraints is directly integrated into the decoding loop and happens on the token level. Compared to text-based high-level APIs, this means you can save a bunch of useless continuations the LM will produce, that further down the pipeline you have to discard, as you may want to enforce constraints, insert some follow-up instruction, or tool execution result.