Hacker News Reader

Comment by hananova | Hacker News Reader

Why do none of these apps allow you to set the system prompt? I find these LLM apps kind of useless without being able to refine the way in which the model will respond to later questions.

View on HN

would love for you to give cnvrs a shot!

- save characters (system prompt + temperature, and a name & cosmetic color) - download & experiment with models from 1b, 3b, & 7b, and quant options q2k, q4km, q6k - save, search, continue, & export past chats

along with smaller touches: - custom theme colors - haptics

and more coming soon!

https://testflight.apple.com/join/ERFxInZg

Do not download this.

I downloaded this on my 14 Pro and it completely locked up the system to the point where even the power button wouldn’t work. I couldn’t use my phone for about 10 minutes.

I've used it for a couple weeks on my 15 Pro and I haven't experienced anything like that. (IMO it's well worth the download)

The developer is also pretty responsive and actively looking for feedback (which is why it's currently free on TestFlight)

I’m very sorry about your experience. That’s definitely not what I was aiming for, and I can imagine that was a nasty surprise. Any hang like that is unacceptable, full stop.

My understanding is Metal is currently causing hangs on devices when there is barely enough RAM to fit the model and prompt, but not quite enough to run. Will work on falling back to CPU to avoid this kind of experience much more aggressively than today.

Thank you for taking the time to both try it out and to share your experience; I will use it to ensure it’s better in the future.

Thanks for the response. Unfortunately on my device the behavior makes it impossible to report a bug using a screenshot as requested in the app. I can give you more device info if you want to narrow down the cause.

Yes of course, I would very much appreciate that, if you’d be so generous — thank you! You can email britt [at] bl3 [dot] dev

That is an iOS bug. No app should be able to do this.

So rather than reporting in the app you can report it in Feedback Assistant, if you want to.

Exactly the same here - full lock up for 2 minutes without being able to reboot even with hardware buttons.

I’m very sorry to hear you had such a poor experience as well. I’m sure it’s little consolation at this point having been inconvenienced as you have — it’s certainly not what I aim for in my work!

I’ve just submitted a new update for review with a number of small but material changes to address these issues: https://news.ycombinator.com/item?id=38920916

I hope you’ll consider giving it another shot once that’s live, and thank you for taking the time not just to test but also to report your experience!

Britt

Thanks. I did test your new version but unfortunately similar issues. App completely hung and entire OS was sluggish. iPhone 13 Pro, iOS 17.1.2. Unfortunately I won’t have time to test any more but very good luck with the project.

Quick follow-up:

I’ve just submitted a new update for review with a few small but hopefully noticeable changes, thanks to your feedback:

1. StableLM Zephyr 3b Q4_K_M is now the built-in model, replacing the Q6_K variant.

2. More aggressive RAM headroom calculation, with forced fallback to CPU rather than failing to load or crashing/hanging in such a nasty fashion.

3. New status indicator for Metal when model is loaded (filled bolt for enabled, vs slashed bolt for disabled.)

4. Metal will now also be enabled for devices with 4GB RAM or less, but only when the selected model can comfortably fit in RAM. Previously, only devices with at least 6GB ever had Metal enabled.

I really appreciate your taking the time to test — the hanging you experienced was unacceptable, and I truly am sorry for the inconvenience. I hope you’ll give it another chance once this update is live, but either way I’m grateful for your help in isolating and eliminating this issue!

Britt

This crashes on almost all models for me and also locked up my phone such that only a full reboot would fix it.

I made a free / mostly open source one for iOS that lets you edit the system prompt

https://chatonmac.com

Amazing! Does it submit any data online ?

No.

I definitely do not want any liability of user-generated content or PII or similar. I have no analytics, besides the standard Apple opt-in crash/reporting (not using any 3rd-party service and not sending anything to my own servers).

It downloads configuration from GitHub and HuggingFace directly. It also has OpenAI integration, directly to their servers via BYOK.

trying this out!

BTW and FYI i need to reduce the font size on my iOS device to be smaller than i like in order to use your add/replace API key key pages. if the font is "larger than normal" i can't see/focus on the box to enter or paste in the API key. just increase your iOS system font size to trigger this. thanks in advance for fixing, will try out the app!

Thanks for the detailed report - will fix asap, along with releasing the macOS v1.0. I've just soft launched this so far but have more to come so please let me know anything else.

Mistral instruct doesn't have system prompt AFAIK. Also llama chat system prompt is very useless in my testing.

It seems to work well on GPT4All (macOS) with system prompts?

Can you link to any doc why it shouldn't work?

It does; and if it's LLaMa 2 7B Chat stock from Facebook, that was a little rushed imho, doesn't seem as baked in.

(GPTs matters but it's _very_ bizarre who it thinks it's coming from)

Mistral Instruct does use a system prompt.

You can see the raw format here: https://www.promptingguide.ai/models/mistral-7b#chat-templat... and you can see how LllamaIndex uses it here (as an example): https://github.com/run-llama/llama_index/blob/1d861a9440cdc9...

So the system prompt is just part of the first prompt in a conversation? How is that different from not having a system prompt?

No, system prompts are surrounded by [INST] [/INST]

No, every user input is surrounded by those tags. Scroll down from that link you posted and read the next two example prompts.

No this isn't the case.

Here's the example from https://www.promptingguide.ai/models/mistral-7b#chat-templat...

<s>[INST] What is your favorite condiment? [/INST] "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"</s> [INST] The right amount of what? [/INST]

Note that the sentence starting "Well, I'm quite partial isn't inside the tag.

This example comes from the official system card created by MistralAI available here: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

You can try it:

   ollama run Mistral "<s>[INST] What is your favorite condiment? [/INST] Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen</s> [INST] The right amount of what? [/INST]"

That's the whole context with two user inputs in the INST tags, and one assistant output between/outside of the tags. They're just simulating the beginning of a conversation. You can see this very clearly in the JSON version in the next code block:

  messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"}
  ]

Yes, but this whole block of text gets passed to the LLM on each call as the conversation history. The [INST] tags tell the LLM which parts were inputs (system instructions) as opposed to outputs.

Well, that's not what a system prompt is. There is a difference between system prompts and regular conversational inputs. https://platform.openai.com/docs/guides/prompt-engineering/t...

In actual fact system prompts are just the instruction tuning prompts that provide instructions formatted in the correct way (ie, with [INST]). It's just text all the way down.

Look into their official page: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

There is no system prompt. Unless Llamaindex or some other sources cite something from mistral, I am inclined to believe they just copied it from llama.

Reply on news.ycombinator.com