activatedgeek

We may be talking about two orthogonal things here. And also to be clear, I don't care about theoretical guarantees either.

Now, min-p is solving for the inadequacies of standard sampling techniques. It is almost like a clever adaptive search which other sampling methods fail at (despite truncations like top-k/top-p).

However, one thing that I noticed in the min-p results was that lower temperatures were almost always better in the final performance (and quite expectedly the inverse for creating writing). This observation makes me think that the underlying model is generally fairly good at ranking the best tokens. What sampling allows us is a margin-for-error in cases where the model ranked a relevant next token not at the top, but slightly lower.

Therefore, my takeaway from min-p is that it solves for deficiencies of current samplers but its success is not in contradiction to the fact that logprobs are bad proxies for semantics. Sampling is the simplest form of search, and I agree with you that better sampling methods are a solid ingredient to extract information from logprobs.

In our paper [1], we find that asking a follow up question like "Is the answer correct?" and taking the normalized probability of "Yes" or "No" token (or more generally any such token trained for) seems to be best bet so far to get well-calibrated probabilities out of the model.

In general, the log-probability of tokens is not a good indicator of anything other than satisfying the pre-training loss function of predicting the "next token." (it likely is very well-calibrated on that task though) Semantics of language are a much less tamable object, especially when we don't quite have a good way to estimate a normalizing constant because every answer can be paraphrased in many ways and still be correct. The volume of correct answers in the generation space of language model is just too small.

There is work that shows one way to approximate the normalizing constant via SMC [2], but I believe we are more likely to benefit from having a verifier at train-time than any other approach.

And there are stop-gap solutions to make log probabilities more reliable by only computing them on "relevant" tokens, e.g. only final numerical answer tokens for a math problem [3]. But this approach kind of side-steps the problem of actually trying to find relevant tokens. Perhaps something more in the spirit of System 2 attention which selects meaningful tokens for the generated output would be more promising [4].

[1]: https://arxiv.org/abs/2406.08391 [2]: https://arxiv.org/abs/2404.17546 [3]: https://arxiv.org/abs/2402.10200 [4]: https://arxiv.org/abs/2311.11829

> While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens.

From here: https://platform.openai.com/docs/guides/reasoning

Veritasium covered this effect in a video [1] for the interested.

[1]: https://www.youtube.com/watch?v=aIx2N-viNwY (2016)

I've been very happy with Astro because it is a good example of low floor and high ceiling software. I can start with plain HTML, make it more flexible with Astro language (still very close to HTML), make authoring easier with Markdown (+ lifestyle extensions from Remark/Rehype), and extend to frameworks like React on a need basis (which I use for some pages where I use maps).

[1]: https://sanyamkapoor.com [2]: https://sanyamkapoor.com/kb/the-stack

Unfortunately, I am yet to feel even close to such a breakthrough. I think very few are fortunate to afford such kind of luxury (as the author alludes to as well). There is always something to deliver for, a deadline to meet (although many would argue deadlines are a forcing constraint); a life waiting to happen. With a tiny bit of envy, I feel very happy and inspired when someone does achieve the "flow" state.

On the subject of "tools" to spur creativity, I have always been skeptical. It feels similar to believing that there is a productivity app right around the corner that will unleash your potential. For me, the only true indicator of my productivity has been actually putting in the _time_, making any kind of progress along a chosen direction and then re-evaluating.

What are fellow readers here doing to _own_ their time?

My main gripe with Ansible is the YAML specification. Ansible chooses to separate the task specification and task execution. Pyinfra chooses to directly expose the Python layer, instead of using slightly ugly magic functions/variables. I like this approach more since it allows standard Pythonic control flow instead of using a new (arguably ugly and more hassle to maintain) grammar.

Excited for Pyinfra!

Here is a blog post that describes it: https://huggingface.co/blog/how-to-generate.

I will warn you though that beam search is typically what you do NOT want. Beam search approximately optimizes for the "highest likely sequence at the token level." This is rarely what you need in practice with open-ended generations (e.g. a question-answering chat bot). In practice, you need "highest likely semantic sequence," which is much harder problem.

Of course, various approximations for semantic alignment are currently in the literature, but still a wide open problem.

In any case, there is at least one work that shows that CoT may not be necessary and biasing the decoding path via logit probabilities is also promising. [1]

One could argue it still doesn't contradict the benefits of CoT, but I suspect there is nothing fundamental about CoT, except that we happened to have been pre-training on sequences that use certain prompts that were easy to conceive from a human's perspective.

[1]: https://arxiv.org/abs/2402.10200

  > Let's think "step by step"!

  > Another tidbit I like about data and prompts that miraculously work.
  > Searching for this phrase resulted in this website (among others),  
  > http://geteasysolution.com, containing many math step-by-step solutions. 
  > How common are they? Quite.

  > Makes you think.

But looks like there are plenty of products coming out in this area, and it has me wondering: what is the actual big picture for enterprises here?

I would assume enterprises employ enough people to write yet another query for whatever use case.

- Is the expectation that in the future, we can bring the flexibility of SQL-like languages to people unfamiliar with SQL?

- Perhaps a salesperson unfamiliar with SQL would like to conduct an analysis. Is the volume and variety of such queries so high that optimizing for the turnaround time from an SQL query designed by data analyst to the salesperson to consume the results is so worthwhile?

Perhaps I am underestimating the scale of the problem but would love some insider perspective here.

It looks like a lot of the tooling is heavily engineered for a set of modern popular LLM-esque models. And looks like llama.cpp also supports LoRA models, so I'd assume there is a way to engineer a pipeline from LoRA to llama.cpp deployments, which probably covers quite a broad set of possibilities.

Beyond llama.cpp, can someone point me to what the broader community uses for general PyTorch model deployments?

I haven't quite ever self-hosted models, and am really keen to do one. Ideally, I am looking for something that stays close to the PyTorch core, and therefore allows me the flexibility to take any nn.Module to production.

[1]: https://github.com/jmorganca/ollama/blob/main/docs/import.md.

[1]: https://www.librarything.com/home

(Just thinking out loud next)

If you allow me to be a little imprecise, guided-generation is prompting "just-in-time" unlike the other kind of prompting where you provide all reference tokens "ahead-of-time". Now there's work [1] out there that shows that smaller models rely much more on prompting than larger models do, i.e. smaller models are more faithful to the tokens in the prompt than the larger models which just do whatever they were going to do anyways.

Your results seem very much in line with this kind of a qualitative result --- you show that CodeGen-350M outperforms CodeGen-6B, and CodeGen-6B outperforms text-davinci-003 using MGD. Smaller models perhaps respond more strongly to certain kinds of prompting strategies than larger models do.

[1]: https://arxiv.org/pdf/2307.13702.pdf

Sure. My concern was not specific to llama-2, and was only using it as a placeholder example of a decent pre-trained base model. Replace it with your favorite base model, which you want to use for guided generation. My question is more fundamental - how does post-hoc guided generation interfere with the potential benefits of instruction-tuning?

> About your second point, the goal is that the model can only generate JSON (for example), which can 100% be done by constraining which output token can and cannot be used.

Mechanistically, yes. I am not arguing that. The whole point is to generate JSON that is "useful".

I was rather concerned about a broader fundamental question - how does post-hoc guided generation interfere with the potential benefits of instruction-tuning?

I am curious, however, for the ones who have played around with such libraries wrapping base LLMs with output structure: do base models like Llama2 work very well? My experience says "hell no!" and you do need a fair bit of instruction-tuning for specific use cases to actually get things to work.

And even then, it seems very counter-intuitive to me that given an instruction-tuned model, post-hoc masking of the state-space during generation then amounts to just changing the generation distribution, and potentially detrimental to instruction-tuning?

Shannon's original paper [2] is often touted to be very parseable, but it is always enlightening to read the same basics many times from multiple sources, especially something as foundational as source coding (and broader information theory) in a more modern language.

I also finally discovered the meaning of "differential" in differential entropy [2], the continuous counterpart of discrete probability masses, something that I always swept under the rug.

[1]: https://www.librarything.com/work/16165301/book/242295611 [2]: https://ieeexplore.ieee.org/document/6773024 [3]: https://sanyamkapoor.com/kb/differential-entropy

Oddly, two tests fail for me with Brave (Version 1.51.118 / Chromium: 113.0.5672.126 (arm64)) on macOS Ventura 13.3.1

- pow([0], [0]) gradient, with "Expected «-Infinity» to be close to «0» (diff: < 0.0000005)"

- xlogy([0], [0.30000001192092896]) gradient with "Expected «0» to be close to «-1.2039728164672852»"

I have not needed an ad-blocker for the past 2 years and hardly see any websites breaking. Often syndicate links break, which is not a big deal for me.

The typography/layouts come from Tailwind Typography [1]. I like their defaults.

[1]: https://tailwindcss.com/docs/typography-plugin

With my most recent iteration, I have started using my website more as a mental decluttering tool than a show-tell medium. The wesbite is a bunch of markdown files spiced up with Next.js and my first attempt at structured navigation. I keep a similar private one.

I've now been rethinking of reorganizing into a flat heirarchy + what I like to call "theme portals" - essentially big picture entrypoints into details. This avoids a long list of scattered blog posts. The orrganization style of clean overarching themes like tech/math etc. isn't working for me.

A small bug in the call-to-action at the bottom of the page - If I upvote only "Stripe" in the "Connect your Shops" section, the call-to-action reads:

> Want to know when Stripe, Stripe, and more integrations are ready?

Voting for greater than one shop resolves this repetition error.