versteegen

Wow, that paper is absolutely damning.

> Design: 1.5 years (elapsed) to come up with a design they believed they could implement.

> Development: 3 person-years (3 people × >1 year each)

> (Note: By comparison, implementing the complete Java language from scratch took the same team 2 person-years.

Specifically, an order n Markov chain such as a transformer, if not otherwise restricted, can have any joint distribution you wish for the first n-1 steps: any extensional property. In which case you have to look at intensional properties to actually draw non-vacuous conclusions.

I would like to comment that there are a lot of papers out there on what transformers can or can't do that are misleading, often misunderstood, or abstract so far from transformers as implemented and used that they are pure theory.

A Transformer with a length n context window implements an order 2n-1 Markov chain¹. That is correct. That is also irrelevant in the real world, because LLMs aren't run for that many tokens (as results are bad). Before it hits that limit, there is nothing requiring it to have any of the properties of a Markov chain. In fact, because the state space is k^n (alphabet size k), you might not revisit a state until generating k^n tokens.

¹ Depending on context window implementation details, but that is the maximum, because the states n tokens back were computed from the n tokens before that. The minimum of course is an order n-1 Markov chain.

Yes, but anyway, LLMs themselves are perfectly capable of backtracking reasoning while sampling is run forwards only, in the same way humans do: by deciding something doesn't work and trying something else. Humans DON'T time travel backwards in time and never have the erroneous thought in the first place.

> the language allowed the aliasing of any chunk of memory to a variable of any type, either a scalar or an array of any rank and dimensions.

Wait a minute, I've seen it stated many times that a primary reason FORTRAN can be better optimised than C is that it doesn't allow aliasing memory as easily as C does (what that means, maybe you can say), and that's why 'restrict' was added to C. On the other hand, C's "strict aliasing rule" allows compilers to assume that pointers of different types don't alias the same memory, which allows optimisations.

Of course they can. It doesn't take that long to learn CUDA/etc and hardware details, read through some manuals and performance guides, do some small projects to solidify that knowledge. What you need is talent, and some months. That's why at university I saw plenty of students pull off amazing projects, and there's nothing eyebrow-raising in the slightest about starting a PhD a year after getting a bachelor's and writing software more sophisticated than most programmers would write in their career.

I think the programming profession overvalues experience over skill. However, when I was young I had no appreciation for what the benefits of experience are... including not writing terrible code.

As a long time emacs user, all of that criticism hits uncomfortably close to home, much as I would like to diss Word...

Shareware tapes collection? Was there really such a thing? If so I would imagine it would be one or two demos per tape?

Thanks for posting some of the history... "You might find even earlier examples" is pretty tongue-in-cheek though. [1], expanded in 2003 into [2], has 12466 citations, 299 by 2011 (according to Google Scholar which seems to conflate the two versions). The abstract [2] mentions that their "large models (with millions of parameters)" "significantly improves on state-of-the-art n-gram models, and... allows to take advantage of longer contexts." Progress between 2000 and 2017 (transformers) was slow and models barely got bigger.

And what people forget about Mikolov's word2vec (2013) was that it actually took a huge step backwards from the NNs like [1] that inspired it, removing all the hidden layers in order to be able to train fast on lots of data.

[1] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, 2000, NIPS, A Neural Probabilistic Language Model

[2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin, 2003, JMLR, A Neural Probabilistic Language Model, https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf

This language seems quite similar to Scallop [1], which was recently posted to HN [2]. Both are extensions of Datalog to arbitrary semirings, meaning that they generalise assigning merely true or false to relational statements, allowing probabilistic reasoning and more (almost arbitrary tagging of statements). Scallop is further focused on being differentiable and being able to integrate Scallop code into a PyTorch function. Both seem to have quite a bit of work put into them and have JIT compilers (Scallop also has a GPU implementation). I like the sound of "I have further modernized Dyna to support functional programming with lambda closures and embedded domain-specific languages." [3]

Going to try it out.

[1] https://www.scallop-lang.org/

[2] https://news.ycombinator.com/item?id=43443640

[3] https://matthewfl.com/research#phd

I'm surprised to hear that that schedule worked well. Even if you woke feeling well-rested, what did you feel like towards the end of the day, at your normal/previous sleep time?

Personally in that situation I would (and do) get plenty of sleep every night and then skip the final night. I find the fatigue from a lot of lost sleep normally doesn't hit me in full until the second day after, and the final-day panic is enough to counteract the lack of sleep.

"Should not be" is too harsh, you should have diversity on a board, not only people from that single industry, but maybe you mean "too many". Presumably the investor-appointees have broad experience with many other companies. The type of people who should know how to hire a CEO.

Thanks, very useful! I don't think I need anything as drastic as intercept-anything, but that is a cool project.

2015, wow, I didn't realise how out of date I am.

And yes, I realise that I probably shouldn't require external storage access anyway, although that allows me to do custom previewing. I could use an external file picker. Well, I have some work to do but it's much less daunting now!

> If your app barely uses any permissions (like TFA's apps), you just need to update the targetSdkVersion in the manifest

And what if you do use permissions for stuff like, say, external storage access, don't you need a recent SDK so you can use the APIs for things like the run-time permission requests now mandatory? But I believe the Android SDK removed support for building with ant years ago. And lets say I'm using a legacy framework with an incredibly contrived ant- (and bash-)based build system which there's no way I'm rewriting. Therefore, I'm in big trouble, am I not? Honest question.

You are missing Reasoning's point (although I think you're right about the Nazis needing to create or appropriate some other people's heritage). 'Aryan' was a linguistic term previously mistakenly used to mean Proto-Indo-European. Hitler didn't personally invent Nazi master-race ideology and look to India, it's a small pseudo-intellectual extension of (European) 19th/early 20th century ethnology and eugenics, and that's a very important distinction to make, to understand how the Nazi came to believe what they did through a product of many often-plausible steps and all-too-common character flaws, not through one man.

If you sort by number of employees you get companies where those employees aren't in R&D divisions.

Likely because they haven't got their own suitable SoTA base models of any other size to build on. DeepSeek V3 is 671B, and DeepSeek-Prover-v1.5 [1] is 7B only, based on DeepSeekMath which is 7B, which is based on DeepSeekCoder-Base-7B-v1.5. Maybe DeepSeek-Coder-V2 (16B and 236B) would be a good start but it's merged into DeepSeek V2.5, and V2.5 is inferior to V3. Or some version of Qwen.

[1] https://github.com/deepseek-ai/DeepSeek-Prover-V1.5

> I reconcile by concluding evolution itself may have been the thing that was intelligently designed

I don't understand why I don't hear anyone else take this position. To me it's obvious and it's hard to find any other way to reconcile them. (But I'm not religious.)

Well, actually there's two quite different things that could be asserted along that line and it's not clear which you meant: either that evolution is a mechanism by which the hand of God can meddle in random occurrences to pick outcomes, or that it's completely hands-off.

But I would state the second one differently: that the universe was set up to allow evolution and all the other systems of natures to create the world today. Evolution itself is ultimately a consequence of physics and statistics.

Good idea but poorly stated.

I'm surprised this isn't getting more attention here. This may well be the future.

These large hybrid transformer-SSM models are faster and more efficient but it seems not really by a huge amount: a big deal but Moore's law progression level not breakthrough level.

I gave it a tricky maths problem (from AIMO) and it spat out about 25k thinking tokens (my estimate), going in circles, which took many minutes, far longer than a better model, until finally figuring out the right answer. Not sure whether that's evidence that it is or isn't weaker when looking beyond recent context.

Unfortunately it doesn't seem to be available yet. Scallop and Lobster are both from UPenn, and the Scallop website says "We are still in the process of open sourcing Scallop," so I assume it's a matter of time.

Wow, I'm currently reading the Scallop paper, so funny to see it posted here!

I really love the concept. This isn't just differentiable neurosymbolic declarative probabilistic programming; Scallop has the flexibility of letting you use various (18 included) or custom provenance semirings to e.g. track "proofs" why a relational fact holds, not just assign it a probability. Sounds cool but I'm still trying to figure out the practicality.

Also worth pointing out that it seems that a lot of serious engineering work has been done on Scallop. It has an interpreter and a JIT compiler down to Rust compiled and dynamically loaded as a Python module.

Because a Scallop program (can be) differentiable it can be used anywhere in an end-to-end learning system, it doesn't have to take input data from a NN and produce your final outputs, as in all the examples they give (as far as I can see). For example you probably could create a hybrid transformer which runs some Scallop code in an internal layer, reading/writing to the residual stream. A simpler/more realistic example is to compute features fed into a NN e.g. an agent's policy function.

The limitation of Scallop is that the programs themselves are human-coded, not learnt, although they can implement interpreters/evaluators (e.g. the example of evaluating expressions).

Nice. Though I couldn't understand those "opening lines" until I read in your Introduction:

> For Land, capitalism begins in Northern Italy around 1500 with "the emerging world of technologists and accountants", the spiral interexcitation of "oceanic navigation and place-value calculation", and zero-unlocked double-entry book-keeping

Fibonacci, amongst many others, played a critical role that highly accelerative technology.

Thanks for saying it. People are far too cynical, and blame everything on bots. The truth is they should be a lot more cynical, and blame everything on human tendencies!

Firstly, "R1 14b quantized"? You mean a quantised DeepSeek-R1-Distill-Qwen-14B? That is Qwen 2.5, it is not DeepSeek v3. Surely they didn't finetune Qwen to add more censorship.

Secondly, most of the censorship is a filter added on top of the model when run through chat.deepseek.com (and I've no idea about system prompt), it is only partially due to the actual model's training data.

Also, I'd rather people didn't paste huge blocks of text into HN comments.

What I think is the most interesting part:

> “When gas spirals into a black hole from afar, it can either flow in the same direction the black hole is rotating, or in the opposite direction. We found that the latter case is more likely to match the multi-year observations thanks to their relatively higher turbulent variability”

The direction the black hole is rotating makes a difference due to frame dragging.

https://en.wikipedia.org/wiki/Frame-dragging

I believe DeepSeek models do split numbers up into digits, and this provides a large boost to ability to do arithmetic. I would hope that it's the standard now.

> in the long run, it’s part of a broader process

Well put. And what is this broader process? Those who say new jobs will always be created as others become obsolete think that the broader process is standing still in stream: job titles change, average wealth and comfort increase, but the structure of civilisation will never change. That we'll all still have jobs and won't become unemployable. Eternal stasis sounds absurd to me.

But I don't assume that the change that happens will be bad. And I don't think all jobs will go. I think we'll end up with a fraction of people employed and well off, such as tradespeople, while the majority will busy themselves with unpaid labour. I already do.

This article is amazing. It explains not just why DeepSeek is so successful, but really indicates that innovators elsewhere will be too: that extensive opportunities exist for improving transformers. Yet few companies do (not just China, but everywhere): incredible amounts are spent just replicating someone else's work with a fear of trying anything substantially different.

Define "by definition".

Because this statement really makes no sense. Transformers are perfectly capable (and capable of perfectly) learning mathematical functions, given the necessary working-out space, e.g. for long division or for algebraic manipulation. And they can learn to generalise from their training data very well (although very data-inefficiently). That's their entire strength!