227 karma · joined January 19, 2013
Anecdotal reports from Europe (one insurance company, two banks, one systems integrator, one consulting firm; all multinational): All values in the risk assessment systems must be reassessed if an American partner is involved. For two companies, this must be done on a monthly basis.
A chancellor in Germany once proclaimed a ‘policy of steady hands’ when the opposition accused him of unsettling the economy with his many reforms.
GPU MMUs can handle multiple line in parallel. But not 10k cores at the same time. The HBM is not able to transfer 3.5TByte sequencial.
> But compute becomes the bottleneck in certain scenarios. With long context sequences, attention computation scales quadratically with sequence length.
Even if the statement about quadratically scales is right, the bottleneck we are talking about is somewhere north by factor 1000. If 10k cores do only simple matrix operations each needs to have new data (up to 64k) available every 500 cycles (let's say). Getting these amount of data (without _any_ collision) means something like 100+GByte/s per core. Even 2+TByte/s on HBM means the bottleneck is the memory transfer rate, by something like 500 times. With collision, we talk about an additional factor like 5000 (last time I've done some tests with a 4090).
US-America has looked at the subject of surveillance of its own population and its own (possible) collapse many times and often in a timely manner.
"The Conversation": https://en.wikipedia.org/wiki/The_Conversation
"Enemy of the State": https://en.wikipedia.org/wiki/Enemy_of_the_State_(film)
"The Siege": https://en.wikipedia.org/wiki/The_Siege
"In the Heat of the Night": https://en.wikipedia.org/wiki/In_the_Heat_of_the_Night_(film...
"Eagle Eye": https://en.wikipedia.org/wiki/Eagle_Eye
If you ask publicly, ‘What if I've seen XYZ?’ then it's actually already too late.
I come from Germany, from East Germany. And some people there wanted to know if you had seen certain films and how often. And ‘Idiocracy’ would have been very high up on their list.
Not all films were banned right from the start (‘The Legend of Paul and Paula’ [1]), but right from the beginning the Stasi found it very interesting who had watched the film.
[1] https://en.wikipedia.org/wiki/The_Legend_of_Paul_and_Paula
> … you can expect the speed to half when going from 4k to 16k long prompt …
> … it did slow down somewhat (from 25T/s to 18T/s) for very long context …
Depends on the hardware configuration (size of VRAM, speed of CPU and system RAM) and llama.cpp parameter settings, a bigger context prompt slows the T/s number significantly but not order of magnitudes.
Facit: gpt-oss 120B on a small GPU is not the proper setup for chat use cases.
Audio, video, screen recordings, etc. from a single customer could be something between 1 and 10 GByte per day on average. After training you might get something like 3 MByte in additional model size per day. Even with 1 billion active users you would need to store additional data with 1 billion GByte (again on hot storage, like expensive GPU memory). The total amount of the memory of GPUs sold by NVIDIA is not even close to 400mio GByte (NVIDIA 3.8mio data center GPUs in 2023).
This is under the assumption that we don't see any significant optimization in the meantime. Looking back over the last eight years, the probability of no progress on the software side is near zero.
For a breakthrough in the consumer market, running LLM on-device with today's capabilities requires solving one key topic: "JIT learning" [2]. We can see some progress here [3, 4]. Perhaps the transformer architecture is not the best for this requirement, but it is hard to argue that it is impossible for Generative AI.
Due to today's technical limitations, we don't have real personal assistants. This could be the Mac for Apple in the AI era.
[1] https://gadgetversus.com/graphics-card/apple-a11-bionic-gpu-...
[2] Increasing context size is not a valid option for my scenario as it also increases the computation demand linear.
[2] https://arxiv.org/abs/2311.06668
[3] https://arxiv.org/abs/2305.18466
[Edit: decimal separator mess]
Is this a reference to the AMD chip, or just a fragment of a removed numbered list?
Edit: It‘s a fragment of the PDF-to-HTML [1]
Only one suggestion: On page "OpenAI-compatible API" it would be great to have also a simple example for the pure REST call instead of the need to import the OpenAI package.
[1] https://de.wikipedia.org/wiki/Halbgefrorenes (german)
[2] https://en.wikipedia.org/wiki/Solyanka (english)
Edit: format
> I study turbulence, shocks, magnetic fields and relativistic particles in the Universe using challenging large cosmological simulations.
[1] https://htxt.co.za/2025/04/iconic-concorde-back-to-the-skies...
The author Dennis Overbye about his retirement in December 2024 [1]
> "Gödel's theorem debunks the most important AI myth. AI will not be conscious"
Same statement from Penrose here with Lex Friedman: "Consciousness is Not a Computation" [1].
In more complex words: imagine a processor which is able to process every instruction in 10 clock cycles. But also the processor is able to get new input for this instruction on every clock cycle and starts to process this new input in a pipeline. After the first input you have to wait ten clock cycles. But if you feed the input line every time you will get the output also permanently.
In the case of GPUs, it is now not only a topic of a single pipeline, but multiple in parallel. Depends on your data and algorithm it can be thousands in parallel.
(Can I access a mmap file, display it in real time with rerun and save a history at the same time?)