Hacker News Reader

Comment by rckrd | Hacker News Reader

Similar strategies with the logitsprocesor. It is a more generalized version that's not just constrained to JSON parsing, but any regex. JSONformer/clownfish try to parse the types syntactically.

A regex is a better fit for a different class of problems. You might implement a JSONformer/clownfish with this instead.

View on HN

Makes sense. I wonder if it would actually make more sense to just plug a function that generates the mask because in general it's just:

  fn complete(input, masker):
    completion = input
    while (not done yet):
      mask = masker(completion)
      completion = complete(complete, mask)
    end
    return completion
  end

is that accurate?

There's probably a better API that wraps generate, but there's a bit more work than the logit mask.

You have to go one token at a time, otherwise the masking becomes combinatoric rather than linear (two tokens at a time -- need to generate all two token pairs, etc.).

But otherwise, that's what the code does! https://github.com/r2d4/rellm/blob/main/rellm/rellm.py#L21

Reply on news.ycombinator.com