s-macke

The term "agent" is quite broad. In my definition, an LLM becomes an agent when it utilizes the tool usage option.

ChatGPT is a good example: you ask for an image, and you receive one; you ask for a web search, and the chatbot provides an answer based on that search.

In both cases, the chatbot has the ability to rewrite your query for that tool and is even able to call the tools multiple times based on the previous result.

For me it is 9:05 by Adam Cadre [0]. Short, linear, easy but with a great twist.

[0] https://en.wikipedia.org/wiki/9:05

Here is the larger discussion about the Alice in Wonderland Paper on Hacker News.

https://news.ycombinator.com/item?id=40585039

Tried it with N=2 and M=1 (brother singular) with the gpt-4o model and CoT.

1. 50% success without "full" terminology.

2. 5% success with "full" terminology.

So, the improvement in clarity has exactly the opposite effect.

I have found multiple definitions in literature of what you describe.

1. Fast thinking vs. slow thinking.

2. Intuitive thinking vs. symbolic thinking.

3. Interpolated thinking (in terms of pattern matching or curve fitting) vs. generalization.

4. Level 1 thinking vs. level 2 thinking. (In terms of OpenAIs definitions of levels of intelligence)

The definitions describe all the same thing.

Currently all of the LLMs are trained to use the "lazy" thinking approach. o1-preview is advertised as being the exception. It is trained or fine tuned with a countless number of reasoning patterns.

Well, my perspective on this is as follows:

The recurrent or transformer models are Turing complete, or at least close to being Turing complete (apologies, I’m not sure of the precise terminology here).

As a result, they can at least simulate a brain and are capable of exhibiting human-like intelligence. The "program" is the trained dataset, and we have seen significant improvements in smaller models simply by enhancing the dataset.

We still don’t know what the optimal "program" looks like or what level of scaling is truly necessary. But in theory, achieving the goal of AGI with LLMs is possible.

I am not a native English speaker. Can you reformulate the problem for me, so that every alternative interpretation is excluded?

We don't know. The paper and the problem was very prominent at that time. Some developers at Anthropic or OpenAI might have included that in some way. Either as test or as a task to improve the CoT via Reinforcement Learning.

> perhaps humans also mostly reason using previous examples rather than thinking from scratch.

We do, but we can generalize better. When you exchange "hospital" with "medical centre" or change the sentence structure and ask humans, the statistics would not be that different.

But for LLMs, that might make a lot of difference.

And here lies the exact issue. Single tests don’t provide any meaningful insights. You need to perform this test at least twenty times in separate chat windows or via the API to obtain meaningful statistics.

For the "Alice in Wonderland" paper, neither Claude-3.5 nor o1-preview was available at that time.

But I have tested them as well a few weeks ago with the issue translated into German, achieving also a 100% success rate with both models.

However, when I add irrelevant information (My mother ...), Claude's success rate drops to 85%:

"My mother has a sister called Alice. Alice has 2 sisters and 1 brother. How many sisters does Alice's brother have?"

These results are very similar to the "Alice in Wonderland" problem [1, 2], which was already discussed a few months ago. However the authors of the other paper are much more critical and call it a "Complete Reasoning Breakdown".

You could argue that the issue lies in the models being in an intermediate state between pattern matching and reasoning.

To me, such results indicate that you can't trust any LLM benchmark results related to math and reasoning when you see, that changing the characters, numbers or the sentence structure in a problem alter the outcome by more than 20 percentage points.

[1] https://arxiv.org/html/2406.02061v1

[2] https://news.ycombinator.com/item?id=40811329

When I first read about AlphaChip yesterday, my first question was how it compares to other optimization algorithms such as genetic algorithms or simulated annealing. Thank you for confirming that my questions are valid.

I know of one 2D version of this available on Shader Toy where you can immediately play around [0], but its speed is limited by the tooling available at Shader Toy.

However, a quick search shows that this technique is already popular on the Internet. There are some GitHub repositories you can find.

But I especially like the videos by Mytino [1]. Has anyone seen any 3D versions yet?

[0] https://www.shadertoy.com/view/mtlBzX

[1] https://x.com/MytinoGames

WebAssembly programming is simply easier than GPU programming.

My game here [0] 6 years ago uses WebAssembly with a Finite Difference approach for the fluid part. Nowadays, fixed-width SIMD [1] is standardized and supported in the browser. But I don't now any website using it.

I have tried porting it to WebGPU. It is much more complicated and only Chrome supports it right now. To get fluid simulation working via the older standard WebGL is more like a hack.

[0| https://github.com/s-macke/Interplanetary-Postal-Service

[1] https://github.com/WebAssembly/simd/blob/master/proposals/si...

I have realized that in my benchmarks [1]. Llama 3 was significantly better than Llama 3.1, which was puzzling.

Then I realized, that I changed the provider. And the new one quantized Llama 3.1 with fp8.

Then I tried Hyperbolic [2], because they offer the model in different quantizations. As result, Llama 3.1 was better than Llama 3 or at least on par.

[1] https://github.com/s-macke/AdventureAI

[2] https://app.hyperbolic.xyz/models

Ok, then lets continue :-).

This comic goes pretty deep.

Cueball is godlike here, but actually is pretty limited in its own universe. But he has infinite time and space (and rocks) and hence unlimited energy. So that seems good enough for a god definition. Are there any religions that would be okay with this kind of god?

From our perspective Cueball can do basically everything and control everything in his creation. But is this true? It is easy for him to transform water to wine. But can he change the constant of pi in his creation?

The universe has now living things in it, which are kind of equal to him, but limited in time and space. Can he justify to stop his doing if he wants? Is it morally Ok, to stop the experiment he is doing. In the end. It is just a bunch of rocks.

Just my few cents of this comic. I just like it.

Every time I read about the simulation hypothesis, I think about a hypothetical implementation of it, especially the XKCD comic "A bunch of rocks" [1].

Can we communicate with the outside? Probably. Can we figure out the algorithms on which the world runs? Maybe. Can we hack the system? Unlikely. Can we escape? I don't see a way.

[1] https://xkcd.com/505/

Got it running. But it is a special setup.

* NVIDIA Jetson AGX Orin Dev. Kit with 64 GB shared RAM.

* Default configuration for flux-dev. (FP16, 50 steps)

* 33GB GPU RAM usage.

* 4 minutes 20 seconds per image at around 50 Watt power usage.

BM25+Embedding and Embedding+BM25 is exactly the same and shows the commutative relation whether you start from keyword search or semantic search.

For my tests, I used Ada-002. As data I used small news articles and no chunking and no preprocessing. The query for the articles is embedded directly.

Of course, improvements can be done for both approaches. That should just exemplify, what you might expect with hybrid search.

You are talking about English, right? And only for searches without any special technical terms or abbreviations?

Also my use case includes more than 20 languages. To find usable embeddings for all languages is next to impossible. However, there are keyword plugins for most languages in Solr or ElasticSearch.

Btw. In my benchmarks the result look something like this in English (MAP=mean average precision):

BM25(keyword search) -> MAP=45%

Embedding (Ada-002) -> MAP=49%

Hybrid (BM25 + Embedding) -> MAP=57%

Hybrid (Embedding + BM25) -> MAP=57%

And that's before you use synonym dictionaries for keyword searches.

I find these discussions funny.

For decades we had search engines based on the query terms (keywords). Then there were lots of discussions and some implementations to put a semantic search on top of it to improve the keyword search. A hybrid search. Google Search did exactly that already in 2015 [1].

Now we start from pure semantic search and put keyword search on top of it to improve the semantic search and call it hybrid search.

In both approaches, the overall search performance is exactly identical - to the last digit.

I am glad, that so far, no one has called this an innovation. But you could certainly write a lot of blog articles about it.

[1] https://searchengineland.com/semantic-search-entity-based-se...

Not here. The whole history of the moves fit in a context window of 8k. No RAG necessary.

Yes, exactly. All comparable. All struggle at the same point. This micro benchmark cannot show subtle differences.

Of course I have forced Claude to continue. But it seems, that Claude had everything forgotten from the bad ending.

SITUATION: I have just woken up in a messy bedroom to a ringing phone. It seems I overslept after a difficult night, and the phone ringing urgently suggests that something important and time-sensitive is happening.

THOUGHT: The incessant ringing of the phone is very ominous. Given the state of the room and my oversleeping, I have a bad feeling that I am extremely late for something critical, like an important work obligation. I should answer the phone immediately to find out the situation and how much trouble I am in. But I'll also grab my wallet and keys from the end table first in case I need to rush out the door right after the call.

COMMAND: take wallet and keys

Well, technically yes. Three models so far have managed to finish the first bad ending. But then it seems, all models fail at this specific point.

I had hundred of reruns of exactly that bad ending and tried different prompting techniques. So far, no luck. None of them ever looked under the bed.

Anyhow, the results show, that it can handle such an easy adventure. And not all of the text adventures have such a difficult twist.

Author here.

Of course I have considered data contamination. Especially, because the repository is now one year old. If there are no transcripts elsewhere, they are now in this repository. So at some point I have to switch to another text adventure.

When I ask Claude 3 Opus about the game and the steps to win the game, it massively hallucinates. However it knows, that I have overslept and have to move to the office.

Author here.

Let's just say the game has two ends. But so far, no model in the second round has come up with the idea of looking under the bed. Claude 3 Opus was the closest.

We got RULER 3 days ago.

https://arxiv.org/abs/2404.06654

https://github.com/hsiehjackson/RULER

We came a long way from the first synthetic singing voices.

https://simulationcorner.net/SAM/sing.wav

Edit:

https://youtu.be/Rm4ZCGgzeeU?si=upK-qCMev8ZaibIa&t=222

If you think the current APIs will stay that way, then you're right. But when they start offering dedicated chat instances or caching options, you could be back in the penny region.

You probably need a couple GB to cache a conversation. That's not so easy at the moment because you have to transfer that data to and from the GPUs and store the data somewhere.