versteegen

516 instances of "\r\n"!

Well unlike most countries Australia and China have a lot of two-way trade (though imports to China are double exports) because China imports lots of raw minerals (as well as food) from them, and noone buys as many raw materials as China does. And thousands of km is still much less than twenty thousand to Europe.

https://en.wikipedia.org/wiki/List_of_the_largest_trading_pa...

Think about what happens when a student solves some math exercises using some maths they don't fully understand, or some example they contrived to test what they read. They might make some false starts, work through some steps, make reference to other ideas that they do understand, and finally find a path to the solution. This is actually incredibly analogous to LLM-generated synthetic training data. And nobody's going to print those crossed-out scrawls in a textbox either! But then the student learns from the chain-of-reasoning which arrived at the result. They learn what steps they should have followed one after the other so they can apply it effortlessly next time, what mathematical tool A on problem type B results in, and so forth. They distill some insights from it. Well LLMs can also learn many of these same things. They certainly don't learn as well or as much as humans (just look at their appalling data requirements), but my point is synthetic training data really does work. So expect training compute used on frontier models to keep scaling.

Terence Tao wrote his comments on this talk here:

https://mathstodon.xyz/@tao/113601397729489852

> Akshay Venktesh gave a thoughtful and accessible talk recently entitled "(Re)imagining mathematics in a world of reasoning machines". I particularly liked his highlighting of Davis and Hersh's pithy definition of mathematics (which I had not been previously aware of) as "the study of mental objects with reproducible properties"; it is a nice way to abstract out the most essential features of the diverse universe of mathematical objects (numbers, shapes, functions, etc.) that mathematicians actually study.

> (My own personal definition of mathematics is more descriptive than prescriptive: mathematics is what mathematicians do, and mathematicians are the people who study mathmatics. This sounds like a tautologically circular definition, but I think of it more as describing the equations of motion of a stochastic dynamical system, in which breakthroughs in mathematics attract more attention by mathematicians, which in turn redefines the relative weighting of different fields of mathematics.)

> It sounds like it will be able to crack some hard math problems, but not actually do mathematics.

What, do you say that because format of the FrontierMath problems is a bit contrived? I think I must misunderstanding you; a benchmark can't rule out something it doesn't test. And saying solving these problems requiring "deep domain expertise" isn't real mathematics sure sounds like a No True Scotsman argument. Why shouldn't o3 be able to produce novel mathematical proofs? It's got plenty of maths textbooks and papers to train on, the same thing mathematicians train on.

> only made possible because some other person invented the abstraction of averages then that was collectively disseminated to everyone in education

What I find most fascinating about the history of mathematics is that basic concepts such as zero and negative numbers and graphs of functions, which are so easy to teach to students, required so many mathematicians over so many centuries. E.g. Newton figured out calculus because he gave so much thought to the works of Descartes.

Yes, I think "new" ideas (meaning, a particular synthesis of existing ones) are essentially inevitable, and how many people come up with them, and how soon, is a function of how common those prerequisites are.

I know I'm at fault for emotively complaining about "uninformed takes" in my comment instead of being substantive, which I regret, and I deserve replies such as this. I'll try harder to avoid getting into these arguments next time.

I wouldn't be an AI researcher if I didn't have "faith" that AI as a goal is worthwhile and achievable and I can make progress. You think this is irrational?

I am actually working to improve the SoTA in mathematical reasoning. I have documents full of concrete ideas for how to do that. So does everyone else in AI, in their niche. We are in an era of low hanging fruit enabled by ML breakthroughs such as large-scale transformers. I'm not someone who thinks you can simply keep scaling up transformers to solve AI. But consider System 1 and System 2 thinking: System 1 sure looks solved right now.

> As long as your field keep coming with less and less realistic predictions and fail to deliver over and over

I don't think we're commenting on the same article here. For example, FrontierMath was expected to be near impossible for LLMs for years, now here we are 5 weeks later at 25%.

You're being pretty harsh, but:

> i think most people have very little conceptualization of their own thinking/cognitive patterns, at least not enough to sensibly extrapolate it onto ai.

Quite true. If you spend a lot of time reading and thinking about the workings of the mind you lose sight of how alien it is to intuition. While in highschool I first read, in New Scientist, the theory that conscious thought lags behind the underlying subconscious processing in the brain. I was shocked that New Scientist would print something so unbelievable. Yet there seemed to be an element of truth to it so I kept thinking about it and slowly changed my assessment.

OHRRPGCE

ARC wasn't designed as a benchmark for LLMs, and it doesn't make much sense to compare them on it since it's the wrong modality. Even a MLM with image inputs can't be expected to do well, since they're nothing like 99.999% of the training data. The fact that even a text-only LLM can solve ARC problems with the proper framework is important, however.

> seems obvious to everyone outside of the field

It's obvious to people inside the field too.

Honestly, these things seem to be less obvious to people outside the field. I've heard so many uninformed takes about LLMs not representing real progress towards intelligence (even here on HN of all places; I don't know why I torture myself reading them), that they're just dumb memorizers. No, they are an incredible breakthrough, because extending them with things like internal thoughts will so obviously lead to results such as o3, and far beyond. Maybe a few more people will start to understand the trajectory we're on.

2012 was when the revolutionaries stormed the bastille and overthrew the old guard. But I say it was 2006 when the revolution started, when the manifesto was published: deep NNs can be trained end-to-end, learning their own features [1]. I think this is when "Deep Learning" became a term of art, and the paper has 24k citations. (Interestingly in a talk a Vector Hinton gave two weeks ago he said his paper on deep learning at NIPS 2006 was rejected because they already had one.)

[1] G. E. Hinton and R. R. Salakhutdinov, 2006, Science, Reducing the Dimensionality of Data with Neural Networks

Well for example I decided to use Emacs Lisp to search through the FreeBasic code for certain constructs so that I could use some elisp functions I'd already written to help edit FB in Emacs. It's throwaway, noone else will ever want to edit (or probably even use) it. FB is niche enough that there isn't much tooling available for it.

I did indeed add Squirrel just to try it out (I'm considering using it as a scripting language), though I wouldn't be surprised if Euphoria and XSLT were also originally used for that reason. XSLT is there for generating .html documentation. C because I needed speed but didn't want to use C++, C++ because a C++ programmer contributed, Obj-C for the Mac port, JS for the web port, x86 assembly for the original DOS version, Bash, Batch and Python for scripts... any project can quickly reach 10 languages this way.

I actually co-maintain a project which has exactly 17 different languages* (though 3 of which are since removed, and 6 languages cover 99% of everything), and noone has ever complained when I add a new language for something throwaway... I think the real issue isn't using multiple languages, it's using a language as intimidating as Rust. Adding some C or C++ or JS to a project usually isn't nearly as problematic (although admittedly it certainly is for the sorts of inexperienced programmers who'd like to contribute to my project).

* If you're curious, in roughly decreasing order of use: FreeBasic/ReloadBasic, HamsterSpeak, Euphoria, C++, Python, C, Bash, Batch, XSLT, JS, Objective C, Emacs Lisp, Squirrel, GDB script. Removed languages: QuickBasic, x86 assembly, Makefile

> That vector has a lot of information in it, it's not a debatable thing.

Encoding the next token is the minimum possible amount of information it might contain; that's not much information (the distribution over the next token is just a projection from the embedding space). E.g. it would be useless for any classification task.

> surely has a rich understanding of more than just the next token it wants to predict

> the last hidden layer is obviously super rich in semantic information

I don't agree that this is obvious, and think it's likely wrong (see the sibling thread [1]). The model has to at some point compress down its prediction for the entire future string of text to a prediction for a single token. There's no prior reason to assume it does this mostly in the final "LM head" linear layer, and the inputs to it don't have to predict anything other than the very next token so there's no reason it should (which is what I think psb217 was getting at), but I'm not familiar with what research has been done into it. On the other hand, processing seems to typically be concentrated in the central layers.

[1] https://news.ycombinator.com/item?id=42379167

Hydroponics usually isn't competing on cost, it's competing on other axes, such as having a greater amount of control. Just the same as greenhouses vs growing in the open. Farming is mostly about dealing with or preventing everything that could go wrong.

Well... if you look at an opcode example, eg [1], it's actually almost entirely ordinary C++ which would only look slightly different if this were a non-JIT-compiling interpreter without Deegen. I say that being moderately familiar with the PUC Lua, LuaJIT, Squirrel and other VM codebases. If this is enough to produce a good JIT, that's incredible. It's quite verbose C++ but that's code style, not due to Deegen. The Deegen bit is a few lines at the bottom, which is DSL-like. I think C++ is a good choice, because it's a good choice for writing a VM.

[1] https://github.com/luajit-remake/luajit-remake/blob/master/a...

-LLVM is huge. This rules it out for many usecases.

-LLVM's API is notoriously unstable. I've probably seen 10 different projects struggling with this problem. They're often tied to a specific version or version range, can't use the system install of LLVM, and bitrot fast. As one person put it, "I mean that there has been an insane amount of hours trying to make llvm as a dependency and that it doesn't break on other devices, one of zig's main complains about llvm." [in thread [1]]. In comparison, gccjit's API, while not as mature, is supposedly simpler, higher level/easier, and more stable [1].

-LLVM was never designed to be used as a JIT compiler. It's slow and the API doesn't sound good for that.

For example for the last couple of months I've been using Julia, and it seems to me that the biggest problem with the language is LLVM! It suffers significantly from all three points above. And that's despite the fact that Julia is probably the best place to use LLVM for JIT compiling, since it cares about performance so much. After all the pain I've seen, I would never use LLVM as a dependency.

[1] https://www.reddit.com/r/ProgrammingLanguages/comments/19etc...

Yes! I've been waiting for a practical tool like this, and would love to write a JIT for Squirrel/Quirrel using it.

But I'm looking through the luajit-remake codebase, and there is still a lot of code. Assuming that the drt and deegen directories are Deegen (however, at lease drt/tvalue.h is clearly part of the VM, not of Deegen):

  > fd . -e h -e cpp | egrep -v "test|thirdparty|deegen|drt" | xargs wc --total=only --lines
  34734
  > fd . -e h -e cpp | egrep -v "test|thirdparty" | xargs wc --total=only --lines
  97629

In comparison, Lua 5.2.4 is 20.3k lines of C and LuaJIT 1.1.5, which is a (comparable?) method JIT compiler, is 22.8k lines of C and 4.8k lines of Lua (for dynasm and JIT support). LuaJIT 2.1 is 74.9k lines of C, 13.7k Lua.

My heart sank at the description of being LLVM based. (I couldn't think of a worse choice for creating a JIT compiler.) Thankfully, they don't use LLVM at runtime! LLVM is only for static compilation of the JIT.

I have wondered whether it's worth storing instruction offsets (from the first instruction) rather than raw instruction pointers to increase cache efficiency, then they could be encoded in just 2 (or at worst 3) bytes. At the cost of an extra register.

Interesting. I haven't compared with 4o or GPT4, but I found DeepSeek 2.5 seems to be better than Claude 3.5 Sonnet (new) at Julia. Although I've seen both Claude and DeepSeek make the exact same sequence of errors (when asked about a certain bug and then given the same reply to their identical mistakes) that shows they don't fully understand the syntax for passing keyword arguments to Julia functions... wow. It was not some kind of tricky case or relevant to the bug. Must have same bad training data. Oops, that's diversion. Actually they're both great in general.

A latent that is never active and hence doesn't (seem to) represent anything. A loss term to reduce the occurrence of that, and if it does happen, push it back to being active sometimes.

Kudos! Was it difficult to get it accepted? I've seen ports rejected.

He wrote something along those lines here [1], which was in reply to a completely different, prototype-quality RISC-V port attempt

> Is the sponsor prepared to sponsor the initial review and integration into the LuaJIT default code base by me?

> Is the sponsor prepared to sponsor the inevitable initial bug fixes and the extra effort for continued maintenance that a new architecture entails?

Also, I should have been clearer about the new GC I linked to: I have not seen Mike say anything about it, and I wouldn't be surprised in the least if he rejects it and (wishes to) write his own, because he's had his own plans for many years. It seems impossible to get anything past him without modification. (I think it's a pity to see someone send a PR with a highly informative commit message and he replaces the body with "Thanks to X. #987")

[1] https://github.com/LuaJIT/LuaJIT/issues/628#issuecomment-716...

Actually there is some progress.

For example there is a new higher-performance GC (https://github.com/LuaJIT/LuaJIT/issues/38#issuecomment-1696...) since a year ago (in fact, at least 3 people over the years have taken a stab at writing a new GC!)

And a full port to (certain flavours of) RISC-V was finished a couple months ago and awaiting merge (https://github.com/LuaJIT/LuaJIT/pull/1267), and might be merged separately into the OpenResty fork (https://github.com/openresty/luajit2/pull/236).

> the goal being to find a novel way to get neural networks to generalize from a few samples

Remove "neural networks". Most ARC competitors aren't using NNs or even machine learning. I'm fairly sure NNs aren't needed here.

> why is the MindsAI solution accepted?

I hope you're not serious. They obviously haven't broken any rule.

ARC is a benchmark. The point of a benchmark is to compare differing approaches. It's not rigged.

Perception is the representation of raw inputs into a form useful for further processing, but it is not a feed-forward computation. You repeatedly re-represent what you see as you keep looking. Particularly something like an ARC puzzle where you have to find a representation that reveals the pattern. That's what my ARC solver is about (I did not finish it for the deadline).

> A very interesting corollary is that the first AGIs might be way better thinkers than humans by default

I agree at least this far. Human System 2 cognition has some very severe limitations (especially working memory, speed, and error rate) which an AGI probably would not have. Beyond fixing those limitations, I agree with François that we shouldn't assume there aren't diminishing intelligence returns to better mental architectures.

Hi! As someone who spent the last month pouring myself into the ARC challenge (which has been lots of fun, thanks so much for creating it), I'm happy to see it made harder, but please make it harder by requiring more reasoning, not by requiring more human-like visual perception! ARC is almost perfect as a benchmark for analogical reasoning, except for the need for lots of image processing as well. [Edit: however, I've realised that perception is representation, so requiring it is a good thing.]

Any plan for more validation data to match the new harder testset?