mmoskal

I think the point is they need several wafers to fit the model. Data parallelism between requires too much bandwidth (they only have so much edge real estate for io) so you have to do pipeline parallelism with say one layer per wafer. No one does it on gpu unless you can’t fit the model in one fast network domain (which is 72 GPUs nowadays for Nvidia). There is a Dwarkesh episode from like two months ago that explains this reasonably accurately.

Also 425 RMB is about $59 so $1 of tokens for $0.017 not $0.13 (the discount rate quoted also seems off).

I believe the App Store policy is you have to have a setting to disable ads. And Uber actually has it (though it has 8 different channels or so, apparently "Uber teen accounts" marketing was added recently).

I used the setting and am not getting Uber ads (only Uber ride notifications).

If you want something with very small pcb footprint and DIY cheap cable try Hack connect https://arcade.makecode.com/hardware/dbg

Grammars work best when aligned with prompt. That is, if your prompt gives you the right format of answer 80% of the time, the grammar will take you to a 100%. If it gives you the right answer 1% of the time, the grammar will give you syntactically correct garbage.

OpenAI is using [0] LLGuidance [1]. You need to set strict:true in your request for schema validation to kick in though.

[0] https://platform.openai.com/docs/guides/function-calling#lar... [1] https://github.com/guidance-ai/llguidance

I had good experience with carefully spaced holes in PCB and a 50 mil header, see https://jacdac.github.io/jacdac-docs/ddk/firmware/jac-connec...

The previous article is in the same issue, in science and technology section. This is how they typically do it - leader article has a longer version in the paper. Leaders tend to be more opinionated.

Consciousness (subjective experience) is possibly orthogonal to intelligence (ability to achieve complex goals). We definitely have a better handle on what intelligence is than consciousness.

Counting to 2^61 probably is.

To actually find a collision in 128b cryptographic hash function it would take closer to 2^65 hashes. Back of the envelope calculations suggest that with Pollard's rho it would cost a few million dollars of CPU time at Hetzner's super-low prices. Not nearly mere mortals budget, but not that far off I guess.

Airplanes are dirty, unsafe and unclean?

I think this is like unsafe - most of your code won’t have it, so you get the benefits of borrow checker (memory safety and race freedom) elsewhere.

This seems way too readable! I think you should remove the character literals in the name of purity.

Also, this is likely way more compact than Brainfuck, as the lambda calculus is written essentially as usual.

And seriously, very cool!

https://github.com/verus-lang/verus is similar tool for Rust (developed by previous heavy users of Dafny).

They are reducing forward pass time from say 1.5ms to 1ms. On bigger model you would likely reduce from 15ms to 14.2ms or something like that.

The sglang and vllm numbers are with cuda graphs enabled.

Having said that, 1B model is an extreme example - hence the 1.5x speedup. For regular models and batch sizes this would probably buy you a few percent.

As mentioned in other comments - in TypeScript which follows this gradual typing there is a number of flags to disable it (gradually so to speak). No reason ty wouldn't do it.

If you change one letter in the prompt, however insignificant you may think it is, it will change the results in unpredictable ways, even with temperature 0 etc. The same is not true of renaming a variable in a programming language, most refactorings etc.

I don't know. I suspect most people rate data centers higher than almond milk...

My understanding is that printing 300 page paperback costs like $2 while 50 pages cost $1.50. However you can clearly claim way more money for the 300 pages so publishers are not interested in short books, business or otherwise.

Very typical for anything with CUDA (they tend to compile everything for 10 different architectures times hundreds of template kernel parameters).

Not sure about ClickHouse though.

To put this into perspective, driving for an hour in an electric car (15kW avg consumption) consumes about as much energy as 50,000 chatgpt queries [0] Running your laptop for an hour would be around 100 queries.

[0] https://epoch.ai/gradient-updates/how-much-energy-does-chatg...

Spec decoding only depends on the tokenizer used. It's transfering either the draft token sequence or at most draft logits to the main model.

Just for some callibration: approx. no one runs 32 bit for LLMs on any sort of iron, big or otherwise. Some models (eg DeepSeek V3, and derivatives like R1) are native FP8. FP8 was also common for llama3 405b serving.

Also ~noone runs h100 at home, ie at batch size 1. What matters is throughput. With 37b active parameters and a massive deployment throughout (per gpu) should be similar to Gemma.

The way I understand it: if the instruction are at the top, the KV entries computed for "content" can be influenced by the instructions - the model can "focus" on what you're asking it to do and perform some computation, while it's "reading" the content. Otherwise, you're completely relaying on attention to find the information in the content, leaving it much less token space to "think".

According to Stack Overflow developer survey [0] Rust is at 12.5%, roughly a half of C# or Java and a quarter of Python. Also more than twice Ruby. So definitely not niche.

[0] https://survey.stackoverflow.co/2024/technology#most-popular...

Just to clarify: simple prefix KV cache doesn't require any special model training. It does require the inference framework to support it, but most do by now.

You can see dramatic improvements in latency and throughput if there is a large shared prefix of the queries.

Memory grows linearly, compute grows quadratically (but with small constant - until ~100k the inference will be still dominated by non-quadratic factors).