Hacker News Reader

Taking a step back, was it necessary to roll your own LLM (or whatever FM) API as opposed to using an off the shelf API

Yes we’re running our own inference server because we do nonstandard things in the decoder loop (think logit masking, other types of sampling, etc).