Hacker News Reader

Comment by skytrue | Hacker News Reader

This isn’t reality. Using ChatGPT this way is fruitless, because there is a system prompt you’re fighting against. I can write a one sentence system prompt for the GPT API that specifies GPT to only spit out JSON, and it works fine. It’s a pretty funny series of image, though.

View on HN

Interesting, so you are saying you have an easier time generating JSON using the single prompt models like text-davinci and text-bison, rather than the chat versions?

Sorry, I was non-specific. If you're using ChatGPT, you're basically using a "product" that OpenAI created. It has specific system prompts and prompt engineering to ensure that it stays on the rails. If you, instead, use the OpenAI API's for GPT-3.5/GPT-4, you aren't beholden to the ChatGPT "product". It's very easy to create a chatbot (not using ChatGPT, the product) that only produces JSON. It's just hard to get ChatGPT to do that same activity.

That's why everybody doing experiments in this space should either 1) be using the OpenAI Playground, or 2) using the API, and not using ChatGPT.

Ah ok, I'm primarily using the API already. One interesting thing is that the GPT-3.5 "product" is much faster, but looks to be using a different model in the request, their encoding model iirc. I wonder if they are now using embeddings to cache results to reduce load on the real models when they can?

They don't mean text-davinci and text-bison, but gpt-3.5-turbo and gpt-4 (and gpt-4-32k). Those are the models powering ChatGPT 3.5/4.

The API for them is already structured conversationally - you don't provide a single prompt to complete, you provide a sequence of prompts of different types ("system", "assistant", "user"); however they mix those together on the backend (some ChatML nonsense?), the models are fine-tuned to understand it.

That's what people mean by "API access to ChatGPT". Same models, but you get to specify system prompts, control the conversation flow, and don't have to deal with dog-slow UI or worry about your conversation getting "moderated" and being blocked with a red warning.

(The models themselves are still trained to refuse certain requests and espouse specific political views, but there isn't a supervisor looking at you, ready to step in and report you to the principal.)

I think you missed the sibling comment where myself and GP have already aligned on this.

Don't need you to explain how the APIs work... and it seems that GPT3.5 UI is doing something else, using the "text-davinci-002-render-sha" model, just look in the browser dev tools. I'm not sure the UI is using anything beyond the smallest context size for GPT4 either, give the output is cut off earlier than 3.5 and it too loses focus after enough messages in a conversation...

I have just verified this exact behavior with Bard.

GPT3.5-turbo/GPT4 is way ahead in instruct tuning and does not require such verbal gymnastics.

Reply on news.ycombinator.com