pg_tiktoken solves the problem of tokenizing text data within a Postgres database. The tiktoken_encode function allows you to tokenize text inputs and returns a tokenized output, making it easier to analyze and process text data for various applications. The tiktoken_count function enables users to return the number of tokens in a text, which is useful for checking text length limits, like those imposed by OpenAI's language models.
Neon CEO here. Happy to answer any questions you might have.