Ask HN: How do GPT models generate variable length outputs?? | Hacker News Reader

Ask HN: How do GPT models generate variable length outputs??

2 pointsnothrowaways7 comments

Traditional ML models generally make predictions with a fixed length output , e.g. text of 100 characters. But gpt models can generate a yes/or no and sometimes they generate longer answers. How is it generated?

View on HN

They generate one token at a time. The model outputs a probability for each possible output token, and the tooling around the model can sample from that distribution in several ways depending how much "creativity" you want to see in the output. Then that new token is added as part of the input, and the model is run again. That's why it can take tenths of a second to generate each token even on hardware with teraflops of processing power.

But how is the length of autoregression be determined?

If each token leads to another, it eventually leads to what commonly comes out as an ‘end token’. So each token has a certain chance of leading to a next token which might, possibly, say to end.

  current_token = <prompt>
  while current_token == true:
    next_token = statistically[current_token][choose_random_from(common_order)]
    current_token = next_token

  s = inf
  for i in common_order:
      p = 1
      x = i
      while true:
          x = statistically[x][common_order[x]]
          if x == false:
              break

          p *= x

      if p < s:
          s = p
          shortest_to_end_start_token = i

Or:

  s = - inf
  ...
  if p > s:
      s = p
      longest_to_end_start_token = i

And then maybe you could sort all tokens by their length (if you wanted to find a length of sub-x).

The dimension of variable 'p' (or its number of bits) could depend on the length of common_order (size of training data).

Or:

  x = statistically[x][common_order[x][-1]] # least probable

The eency weency ... got washed out.

Reply on news.ycombinator.com