56.37 tps / 19.34 tps ≈ 2.9
This explains why Opus 4.1 is 3 times the price of Opus 4.5.
420 karma · joined January 10, 2025
56.37 tps / 19.34 tps ≈ 2.9
This explains why Opus 4.1 is 3 times the price of Opus 4.5.
> (~1TB / SSD_read_speed + computation_time_per_chunk_in_RAM) = a few minutes per ~word or punctuation.
You have to divide SSD read speed by the size of the active parameters (~16GB at 4 bit quantization) instead of the entire model size. If you are lucky, you might get around one token per second with speculative decoding, but I agree with the general point that it will be very slow.- Amazon Bedrock serves Claude Opus 4.5 at 57.37 tokens per second: https://openrouter.ai/anthropic/claude-opus-4.5
- Amazon Bedrock serves gpt-oss-120b at 1748 tokens per second: https://openrouter.ai/openai/gpt-oss-120b
- gpt-oss-120b has 5.1B active parameters at approximately 4 bits per parameter: https://huggingface.co/openai/gpt-oss-120b
To generate one token, all active parameters must pass from memory to the processor (disregarding tricks like speculative decoding)
Multiplying 1748 tokens per second with the 5.1B parameters and 4 bits per parameter gives us a memory bandwidth of 4457 GB/sec (probably more, since small models are more difficult to optimize).
If we divide the memory bandwidth by the 57.37 tokens per second for Claude Opus 4.5, we get about 80 GB of active parameters.
With speculative decoding, the numbers might change by maybe a factor of two or so. One could test this by measuring whether it is faster to generate predictable text.
Of course, this does not tell us anything about the number of total parameters. The ratio of total parameters to active parameters can vary wildly from around 10 to over 30:
120 : 5.1 for gpt-oss-120b
30 : 3 for Qwen3-30B-A3B
1000 : 32 for Kimi K2
671 : 37 for DeepSeek V3
Even with the lower bound of 10, you'd have about 800 GB of total parameters, which does not fit into the 512 GB RAM of the M3 Ultra (you could chain multiple, at the cost of buying multiple).But you can fit a 3 bit quantization of Kimi K2 Thinking, which is also a great model. HuggingFace has a nice table of quantization vs required memory https://huggingface.co/unsloth/Kimi-K2-Thinking-GGUF
There were other problems, such as the research cluster of my university not having Docker, but that is a different issue.
https://en.wikipedia.org/wiki/Floating-point_arithmetic#Accu...
> Fail2ban was struggling to keep up: it ingests the Nginx access.log file to apply its rules but if the files keep on exploding…
> [...]
> But I don’t want to fiddle with even more moving components and configuration
You can configure nginx to do rate-limiting directly. Blog post with more details: https://blog.nginx.org/blog/rate-limiting-nginxOf course we do. Just run a benchmark with Cerebras/Groq and compare to the results produced in a trusted environment. If the scores are equal, the model is is either unquantized, or quantized so well that we can not tell, in which case it does not matter.
For example, here is a comparison of different providers for gpt-oss-120b, with differences of over 10 % for best and worst provider.
https://artificialanalysis.ai/models/gpt-oss-120b/providers#...
Unfortunately, the MeGoo OS was discontinued shortly after. https://en.wikipedia.org/wiki/MeeGo
I also found a Python library for fetching YouTube video transcripts, but some issue mentioned that they got banned, so out of caution, I implemented my summary script as a JavaScript bookmarklet instead. It will probably break on the next YouTube update, so I am not sure how useful it is. Also, you have to set your own API key (and maybe URL). I used Groq (not to be confused with Grok), because it is free and very fast.
javascript:(function(){
var GROQ_API_KEY = "YOUR_API_KEY_HERE";
var btn = [...document.querySelectorAll('button')].find(b => b.textContent.trim() === 'Show transcript');
btn.click();
function checkTranscriptAvailable(){
var transcript = document.querySelector('[target-id="engagement-panel-searchable-transcript"]').innerText;
console.log("transcript:", transcript.slice(0, 50));
var length = transcript.replace(/\s/g, '').length;
if (length > 100){
fetch("https://api.groq.com/openai/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": "Bearer " + GROQ_API_KEY,
"Content-Type": "application/json"
},
body: JSON.stringify({
"model": "openai/gpt-oss-120b",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Briefly summarize this transcript:\n\n" + transcript,
},
]
}
]
})
})
.then(res => res.json())
.then(data => alert(data.choices[0].message.content))
.catch(err => alert(err));
}else{
setTimeout(checkTranscriptAvailable, 1000);
}
};
checkTranscriptAvailable();
})();- Games (400GB for Ark, 235GB for Call of Duty, 190GB for God of War)
- LLMs (e.g. DeepSeek-V3.2-Exp at 690GB or Kimi-K2 at 1030GB unquantized)
- Blockchains (Bitcoin blockchain approaching 700GB)
- Deep learning datasets (1.1PB for Anna's Archive, 240TB for LAION-5B at low resolution)
- Backups
- Online video processing/storage
- Piracy (Torrenting)
Of course you can download those things on a slower connection, but I imagine that it would be a lot nicer if it went faster.
> Half the bugs you chase aren’t in your code. They’re in your head.
> You’re not asking for their input, necessarily. You’re asking to have someone listen to you so you can think straight.
> The truth is in the path, not the punchline.
> The log is not your diary. It’s your surveillance system.
> Debugging isn’t just thinking. It’s re-thinking
> These 10 tips aren’t a checklist: they are a starter kit.
> Logs are your sidekick, not your saviour.
def main():
a, b, c = Node("a"), Node("b"), Node("c")
edges = [
a <- b | "first edge",
b <- c | "second edge",
c <- a | "third edge",
]
print("Edges:")
for edge in edges:
print(edge)
class Node:
def __init__(self, name):
self.name = name
def __lt__(self, other):
return Edge(self, other)
def __neg__(self):
return self
def __or__(self, label):
self.label = label
return self
class Edge:
def __init__(self, a, b):
self.a = a
self.b = b
def __repr__(self):
return f"{self.a.name} <- {self.b.name}: {self.b.label}"
main()
It works like this:The '-' sign is interpreted as a unary minus, which is simply discarded.
The '<' symbol is handled by the '__lt__' overload of the Node class.
The '|' operator has precedence over '<' in Python, so the edge label is stored in the right-hand side node.
That being said, I will haunt you if you actually use this in production.
For inference, the DGX Spark does not look like a good choice, as there are cheaper alternatives with better performance.
https://news.ycombinator.com/item?id=45456502 (0 karma)
https://news.ycombinator.com/item?id=45460800 (1 karma)
https://news.ycombinator.com/item?id=45456941 (4 karma)
https://www.forbes.com/sites/kashmirhill/2013/07/24/blueprin...
At the top of the README of the GitHub repository, there are a few links to demos where you can try the model.
> It seems it needs around a $2,500 GPU
You can get a used RTX 3090 for about $700, which has the same amount of VRAM as the RTX 4090 in your ChatGPT response.
But as far as I can tell, quantized inference implementations for this model do not exist yet.
https://nvidianews.nvidia.com/news/nvidia-announces-financia...
This also explains why NVIDIA will not sell high VRAM consumer GPUs: it would cannibalize on their exorbitant data-center profits.
Oops, I did not read that before going ham in the editor. It seems that the files are stored inside the emscripten file system, so they are not lost. I could download my exported 'test.stl' with the following JavaScript code:
var data = FS.readFile('test.stl');
var blob = new Blob([data], { type: 'application/octet-stream' });
var a = document.createElement('a');
a.href = URL.createObjectURL(blob);
a.download = 'test.stl';
a.click();