throwdbaaway

It is the reasoning. During the reasoning process, the top few tokens have very similar or even same logprobs. With gpt-oss-120b, you should be able to get deterministic output by turning off reasoning, e.g. by appending:

    {"role": "assistant", "content": "<think></think>"}

Of course, the model will be less capable without reasoning.

Somehow that article totally ignored the insane pricing of cached input tokens set by Anthropic and OpenAI. For agentic coding, typically 90~95% of the inference cost is attributed to cached input tokens, and a scrappy China company can do it almost for free: https://api-docs.deepseek.com/news/news0802

> But I did not expect something as core as the service managing the leases on the physical EC2 nodes to not have recovery procedure.

I guess they don't have a recovery procedure for the "congestive collapse" edge case. I have seen something similar, so I wouldn't be frowning at this.

A couple of red flags though:

1. Apparent lack of load-shedding support by this DWFM, such that a server reboot had to be performed. Need to learn from https://aws.amazon.com/builders-library/using-load-shedding-...

2. Having DynamoDB as a dependency of this DWFM service, instead of something more primitive like Chubby. Need to learn more about distributed systems primitives from https://www.youtube.com/watch?v=QVvFVwyElLY

Actually I am keen to know how Roblox got impacted. Following the terrible Halloween Outage in 2021, they posted 2 years ago about migrating to a cell based architecture in https://corp.roblox.com/newsroom/2023/12/making-robloxs-infr...

Perhaps some parts of the migration haven't been completed, or there is still a central database in us-east1

This looks impressive. As someone who is not familiar with ML, I do have a question -- surely in 2025 there must be a way to schedule a large pytorch job across multiple k8s clusters? EKS and GKE already provide VPC native flat network by default .

Huh I just moved to AMDVLK yesterday, after learning that it has 50% more PP on llama.cpp compared to RADV: https://www.reddit.com/r/LocalLLaMA/comments/1nabcek/comment...

That makes sense, thanks for the info. Here's a quick recap of the recent MoE models based on the criteria..

correct activated params:

  * DeepSeek V3/R1 series
  * Kimi K2
  * GPT-OSS series

undercount activated params:

  * GLM-4.5 series

overcount activated params:

  * DeepSeek V2 series
  * Qwen3 series
  * Ernie 4.5 series
  * Hunyuan A13B

So GLM-4.5 series omits the embedding layer and the output layer when counting both the total parameters and the active parameters:

> When counting parameters, for GLM-4.5 and GLM-4.5-Air, we include the parameters of MTP layers but not word embeddings and the output layer.

This matches with the calculation I did for GLM-4.5 (355B A32B):

    In [14]: 356732107008 - (775946240 * 2) # token_embd / output are 775946240 each. assume omitted
    Out[14]: 355180214528

    In [15]: 356732107008 - 339738624000 - (775946240 * 2) # parameters that are always active
    Out[15]: 15441590528

    In [16]: 339738624000 * 8 / 160 # parameters from activated experts
    Out[16]: 16986931200.0

Meanwhile, GPT OSS series includes both the embedding layer and the output layer when counting the total parameters, but only includes the output layer when counting the active parameters:

> We refer to the models as “120b” and “20b” for simplicity, though they technically have 116.8B and 20.9B parameters, respectively. Unembedding parameters are counted towards active, but not embeddings.

And Qwen3 series includes both the embedding layer and the output layer when counting both the total parameters and the active parameters.

Why there is no standard in counting? Which approach is more accurate?

You support logprobs, that's wonderful! Fireworks, Synthetic, (ik_)llama.cpp, now I have a quorum.

> use repomap tools on dependencies if creating new code that leverages those dependencies, and have that in context for that work.

It seems to me that currently there are 2 schools of thought:

1. Use repomap and/or LSP to help the models navigate the code base

2. Let the models figure things out with grep

Personally, I am 100% a grep guy, and my editor doesn't even have LSP enabled. So, it is very interesting to see how many of these agentic tools do exactly the same thing.

And Claude Code /init is a great feature that basically writes down the current mental model after the initial round of grep.

https://llm-tracker.info/_TOORG/Strix-Halo has very comprehensive test results for running llama.cpp with Strix Halo. This one is particularly interesting:

> But when we switch to longer context, we see something interesting happen. WMMA + FA basically loses no performance at this longer context length!

> Vulkan + FA still has better pp but tg is significantly lower. More data points would be better, but seems like Vulkan performance may continue to decrease as context extends while the HIP+rocWMMA backend should perform better.

lhl has also been sharing these test results in https://forum.level1techs.com/t/strix-halo-ryzen-ai-max-395-..., and his latest comment provides a great summary of the current state:

> (What is bad is that basically every single model has a different optimal backend, and most of them have different optimal backends for pp (handling context) vs tg (new text)).

Anyway, for me, the greatest thing about the Strix Halo + llama.cpp combo is that we can throw one or more egpu into the mix, as echoed by level1tech video (https://youtu.be/ziZDzrDI7AM?t=485), which should help a lot with PP performance.

With GLM-4.5, we can disable thinking by appending /nothink to the user message. Is that how we suppose to use Octofriend, e.g. "Plan looks good. Go ahead and implement feature X. /nothink"

Actually, you can combine them. When compared to Mac Studio, the main advantage of these Strix Halo boxes is that you still add a bunch of egpu over usb4/oculink, for better PP.

> how the scout -> maverick -> behemoth doesn't scale sparsity according to any formula (less sparse -> sparse -> less sparse)

Ah I see. I didn't notice that behemoth has the same sparsity as scout. That seems quite random indeed.

There is no way that gpt-oss-120b can beat the much larger Kimi-K2-Instruct, Qwen3 Coder/Instruct/Thinking, or GLM-4.5. How did you arrive at this rather ridiculous conclusion? The current sentiment in r/LocalLLaMA is that gpt-oss-120b is around Llama-4 Scout level. But it is indeed the best in refusal.

I thought Kimi K2 uses 8 active experts out of 384? Sparsity should be 48:1. Indeed Llama4 Maverick is the only one that has 128:1 sparsity.

> The bubble itself becomes increasingly intellectually dishonest, increasingly unserious, as it inflates.

You started to sound like Dario, who likes to accuse others as intellectually dishonest and unserious. Anyway, perhaps the strict wage structure of Anthropic will be its downfall in this crazy bubble?

Another problem is that it works like a slot machine -- sometimes the code is good, most of the time the code is mediocre and full of bugs.

Last Friday I spent about $15 in 1 hour using claude code with API key, and the code doesn't really work, even though all the unit tests passed. I am not going to touch it for weeks, while the loss is fresh in my mind.

With a subscription though, you can keep on gambling, until you get a hit.

> The models are then trained to (hopefully) treat the system prompt delimited tokens as more influential on how the rest of the input is treated.

I can't find any study that compares putting the same initial prompt in the system role versus in the user role. It is probably just position bias, i.e. the models can better follow the initial input, regardless of whether it is system prompt or user prompt.

I just learnt the `nl -ba` trick from Codex. Claude Code is most likely doing the same.

As shared by Simon in https://news.ycombinator.com/item?id=44176523, a better agent will prepend the line numbers as a workaround, e.g. Claude Code:

    54 def dicts_to_table_string(
    55     headings: List[str], dicts: List[Dict[str, str]]
    56 ) -> List[str]:
    57     max_lengths = [len(h) for h in headings]
    58 
    59     # Compute maximum length for each column
    60     for d in dicts:

> Here's the technical takeaway: Never use CASCADE deletes on critical foreign keys. Set them to NULL or use soft deletes instead. It's fine for UPDATE operations, but it's too dangerous for DELETE ones. The convenience of automatic cleanup isn't worth the existential risk of chain reactions.

I actually agreed 100% with this learning, especially the last sentence. The younger me would write a long email to push for ON DELETE CASCADE everywhere. The older me doesn't even want to touch terraform, where an innocent looking update can end up destroying everything. I will rather live with some orphaned records and some infra drifts.

And still I got burnt few months ago, when I inadvertently triggered some internal ON DELETE CASCADE logic of Consul ACL.

(I do agree with your other points)

It is possible that the tasks you gave to the model previously were just about easy enough for it to handle, while the few failing tasks you gave recently were a bit too tough for the model, thus it had to cheat.

For the exact same task, some changes in the system prompt used by Claude Code, and/or how it constructs the user prompt, can quite easily make the task either easy enough or not. It is a fine line.

Ah right, my bad. Somehow I thought the prompt was only:

    Say just 'hi'

while the "without any extra words or explanations" part was for the readers of your comment. Perhaps kubb also made a similar mistake.

I used empty system prompt.

o3-mini-2025-01-31 with high reasoning effort replied with "Hi" after 448 reasoning tokens.

gpt-4.5-preview-2025-02-27 replied with "Hi!"

For the code review use case, maybe can try to create the diff with something like `git diff -U99999`, and then send only the diff.

This prompt works fine with Qwen2.5-Coder-32B-Instruct-Q4_K_M:

    Add a line number prefix to each line, stopping at line 27. What's on line 27 of this program?

https://github.com/google/mysql-tools/tree/master/old/mysql-... - Google used to do that too

With gentoo, if you allocate let's say 20G to / on ext4, then you can quite easily run into this issue.

/usr/src/linux will use about 30% of the space and 10% of the inodes.

/var/db/repos/gentoo will use about 4% of the space and 10% of the inodes.

Next you clone the firefox-source hg repo, which will use about 15% of the space and 80% of the inodes.

I am running the base model of Qwen2.5-Coder-32B with llama.cpp. It can only do completion, it can't chat. Where did you get that information from?