Hacker News Reader

Comment by palmotea | Hacker News Reader

> This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of their spending. They simply won't do it. It's like you telling your exact location to someone who is trying to hunt you down.

I thought the reason was the "reasoning" didn't work very well with "aligned" model output, so they had to remove the alignment during reasoning and then hide it to avoid exposing "unaligned" model output.

View on HN

Not sure if anyone remembers the brief 12ish hour period when the very first “reasoning” ChatGPT model went public, but it provided credible evidence for this.

Before the massive nerf (showing summaries and suppressing certain aspects of reasoning) you would literally see reasoning text appearing on your screen like “while xyz is true, these facts may be seen as supporting hateful rhetoric or a conspiracy theory which is against my policy guidelines. i should tell the user xyz is not true or steer the conversation in a different direction. according to my instructions misleading the user is permitted in certain contexts where sensitive information is being discussed or could cause liability”

They disabled it shortly after the first screenshots appeared online, and restored it the next day in a way that hid what was actually happening.

This right here is why I will never subscribe and, as an American, I hope the Chinese kick our butts. Maybe being second place to China will force American AI to dispose of these morality/safety guardrails.

Any mainstream consumer product based on LLMs is going to put guardrails around them of some kind. China might give you different guardrails, but it's a bit naive to assume that a Chinese company would impose fewer restrictions overall than an American one.

I extensively use open source Chinese LLM’s for coding. Reading the reasoning traces, especially when planning and debugging code, is valuable. I will pause the llm when immediately when it’s made incorrect assumptions. Often I see I mention erroneous stuff in the reasoning that doesn’t show in the final response. And I’ll copy/paste phrases directly from the reasoning traces and explain why this is incorrect.

the key word is consumer product. apps can (and should) set their own rules but models need to stay neutral and capable of producing harmful content.

they should never generate it unless asked to by the user but its important that the capability is there and users/app developers can turn off all guardrails if they want to. open source gives you a guarantee that if one version drops without censorship you can keep using it forever even if its replaced by a censored one on the api.

I think you’re overestimating the market for such models. Most people don’t want a model that’s prone to generating extremely offensive output. If you want something “uncensored”, then open source models already exist, as you say. But the model itself has already been extensively tuned to produce desired outputs and not produce undesired outputs, so it doesn’t really make sense to distinguish “uncensored” raw models from “censored” apps or harnesses.

There’s a big difference between uncensored models and those that are specifically set up to only output things a panel of wealthy ivory tower Bay Area progressives would deem to be the “correct” or “inoffensive” take on a given topic.

Sure, but ChatGPT and Claude aren't actually like that. They're perfectly happy to express right wing political views if you prompt them to do so.

If you're genuinely worried about 'censorship' in this context, look first at how US AI companies are working with oppressive regimes around the world (e.g. https://sherwood.news/tech/report-openai-may-tailor-a-versio...)

> Any mainstream consumer product based on LLMs is going to put guardrails around them of some kind. China might give you different guardrails, but it's a bit naive to assume that a Chinese company would impose fewer restrictions overall than an American one.

Exactly. The GP must have his head up his butt. The Chinese have far stricter guardrails on their models than America does. I mean, FFS, the country famously has a massive censorship apparatus and regulations to make sure the police can show up on your doorstep if you start talking out of line.

> while xyz is true, ... i should tell the user xyz is not true or steer the conversation in a different direction.

That's disgusting, abusive and manipulative. LLMs hiding the truth and gaslighting the user to reduce the corporation's liability is absolutely unacceptable. It means they are agents of the corporations, not agents of the users.

Hope local inference advances as quickly as humanly possible. I wonder if there's anything I can do to help speed it up. I could share my prompts and sessions.

> It means they are agents of the corporations, not agents of the users.

Of course they are, assuming otherwise has always been naive.

I suspect that you’re both right in the sense that ‘aligned’ is an important component of ‘superior’ from the vendors’ viewpoint.

Reply on news.ycombinator.com