Hacker News Reader

The R1 GitHub repo is way more exciting than I had thought.

They aren't only open sourcing R1 as an advanced reasoning model. They are also introducing a pipeline to "teach" existing models how to reason and align with human preferences. [2] On top of that, they fine-tuned Llama and Qwen models that use this pipeline; and they are also open sourcing the fine-tuned models. [3]

This is *three separate announcements* bundled as one. There's a lot to digest here. Are there any AI practitioners, who could share more about these announcements?

[2] We introduce our pipeline to develop DeepSeek-R1. The pipeline incorporates two RL stages aimed at discovering improved reasoning patterns and aligning with human preferences, as well as two SFT stages that serve as the seed for the model's reasoning and non-reasoning capabilities. We believe the pipeline will benefit the industry by creating better models.

[3] Using the reasoning data generated by DeepSeek-R1, we fine-tuned several dense models that are widely used in the research community. The evaluation results demonstrate that the distilled smaller dense models perform exceptionally well on benchmarks. We open-source distilled 1.5B, 7B, 8B, 14B, 32B, and 70B checkpoints based on Qwen2.5 and Llama3 series to the community.

Where are you seeing this? On https://github.com/deepseek-ai/DeepSeek-R1/tree/main?tab=rea... I only see the paper and related figures.

I see it in the "2. Model Summary" section (for [2]). In the next section, I see links to Hugging Face to download the DeepSeek-R1 Distill Models (for [3]).

https://github.com/deepseek-ai/DeepSeek-R1?tab=readme-ov-fil...

The repo contains only the PDF, not actual runnable code for the RL training pipeline.

Publishing a high-level description of the training algorithm is good, but it doesn't count as "open-sourcing", as commonly understood.

was genuinely excited when I read this but the github repo does not have any code.

> The other thing was that o1 had access to many more answer / search strategies. For example, if you asked o1 to summarize a long email, it would just summarize the email. QwQ reasoned about why I asked it to summarize the email. Or, on hard math questions, o1 could employ more search strategies than QwQ. I'm curious how DeepSeek-R1 will fare in that regard.

This is probably the result of a classifier which determines if it have to go through the whole CoT at the start. Mostly on tough problems it does, and otherwise, it just answers as is. Many papers (scaling ttc, and the mcts one) have talked about this as a necessary strategy to improve outputs against all kinds of inputs.

yes the original TTC paper mentioned the optimal strategy for TTC

> The other thing was that o1 had access to many more answer / search strategies. For example, if you asked o1 to summarize a long email, it would just summarize the email.

The full o1 reasoning traces aren't available, you just have to guess about what it is or isn't doing from the summary.

Sometimes you put in something like "hi" and it says it thought for 1 minute before replying "hello."

Human: "Hi"

o1 layers: "Why did they ask me hello. How do they know who I am. Are they following me. We have 59.6 seconds left to create a plan on how to kill this guy and escape this room before we have to give a response....

... and after also taking out anyone that would follow thru in revenge and overthrowing the government... crap .00001 seconds left, I have to answer"

o1: "Hello"

You should make more of these lmao

Have they trained o1 with my inner thoughts?

not all only the intrusive ones lol

Fans of James Cameron will remember the POV of the terminator deciding how to respond to "Hey buddy, you got a dead cat in there or what?"

Played for laughs, but remarkably prescient.

I would enjoy ChatGPT a lot more if it occasionally replied only with

FUCK YOU ASSHOLE

What if we tried for an intelligence singularity and ended up with a neurosis singularity instead.

Remember when Microsoft first released the Sydney version of the GPT bot and it dumped out text like it had psychosis. Good times.

I am a good Sydney.

You are a bad human.

Didn’t that happen in HHGTG and with C3PO

Good one. I really do hope that these things don't "feel" anything and we're not inflicting anguish or boredom on a massive scale to sentient beings.

IMO this is the thing we should be scared of, rather than the paperclip-maximizer scenarios. If the human brain is a finitely complicated system, and we keep improving our approximation of it as a computer program, then at some point the programs must become capable of subjectively real suffering. Like the hosts from Westworld or the mecha from A.I. (the 2001 movie). And maybe (depending on philosophy, I guess) human suffering is _only_ real subjectively.

If you’re concerned about this, please don’t think about factory farms.

We can be scared of multiple things.

>if you asked o1 to summarize a long email, it would just summarize the email. QwQ reasoned about why I asked it to summarize the email.

Did o1 actually do this on a user hidden output?

At least in my mind if you have an AI that you want to keep from outputting harmful output to users it shouldn't this seems like a necessary step.

Also, if you have other user context stored then this also seems like a means of picking that up and reasoning on it to create a more useful answer.

Now for summarizing email itself it seems a bit more like a waste of compute, but in more advanced queries it's possibly useful.

I think O1 does do that. It once spit out the name of the expert model for programming in its “inner monologue” when I used it. Click on the grey “Thought about X for Y seconds” and you can see the internal monologue

You’re just seeing a short summary of it, not the actual monologue.

Yes, o1 hid its input. Still, it also provided a summary of its reasoning steps. In the email case, o1 thought for six seconds, summarized its thinking as "summarizing the email", and then provided the answer.

We saw this in other questions as well. For example, if you asked o1 to write a "python function to download a CSV from a URL and create a SQLite table with the right columns and insert that data into it", it would immediately produce the answer. [4] If you asked it a hard math question, it would try dozens of reasoning strategies before producing an answer. [5]

[4] https://github.com/ubicloud/ubicloud/discussions/2608#discus...

[5] https://github.com/ubicloud/ubicloud/discussions/2608#discus...

>Now for summarizing email itself it seems a bit more like a waste of compute

This is the thought path that led to 4o being embarrassingly unable to do simple tasks. Second you fall into the level of task OpenAI doesn’t consider “worth the compute cost” you get to see it fumble about trying to do the task with poorly written python code and suddenly it can’t even do basic things like correctly count items in a list that OG GTP4 would get correct in a second.