Hacker News Reader

Comment by layer8 | Hacker News Reader

> Thought process

Given that the language of the thought process can be different from the language of conversation, it’s interesting to consider, along the lines of Sapir–Whorf, whether having LLMs think in a different language than English could yield considerably different results, irrespective of conversation language.

(Of course, there is the problem that the training material is predominantly English.)

View on HN

I’ve wondered about this more generally (ie, simply prompting in different languages).

For example, if I ask for a pasta recipe in Italian, will I get a more authentic recipe than in English?

I’m curious if anyone has done much experimenting with this concept.

Edit: I looked up Sapir-Whorf after writing. That’s not exactly where my theory started. I’m thinking more about vector embedding. I.e., the same content in different languages will end up with slightly different positions in vector space. How significantly might that influence the generated response?

The answer is yes, LLMs have different behavior and factual retrieval in different languages.

I had some papers about this open earlier today but closed them so now I can't link them ;(

I just tried your experiment, first asking for a bolognese sauce recipe in English, then translating the prompt to Italian and asking again. The recipes did contain some notable differences. Where the English version called for ground beef, the Italian version used a 2:1 mix of beef and pancetta; the Italian version further recommended twice as much wine, half as much crushed tomato, and no tomato paste. The cooking instructions were almost the same, save for twice as long a simmer in the Italian version.

More authentic, who knows? That's a tricky concept. I do think I'd like to try this robot-Italian recipe next time I make bolognese, though; the difference might be interesting.

FWIW, and tangential, the biggest (and time consuming) difference I ever found in making bolognese was hand cutting the meat instead of getting it ground.

The texture was way better. It's a pain to do (obviously) but worth trying at least once, IMO.

Thanks for the recommendation. Diced pancetta is readily available here, but I'd have to chop up the beef myself; which cut did you use?

Recipe calls for skirt steak or chuck. I used chuck. Skirt steak would probably taste nicer, though, but might also be harder to chop.

I ended up chopping it down to 2-3mm (~1/8in?) bits, and it helps to have the meat really cold (eg having hung out in the freezer for a bit).

The italian counterpart of what english speakers call "bolognese sauce" would be "ragù alla bolognese". I've never heard anyone call it "salsa bolognese", it's mostly called "ragù" only as it's most common type.

Nonetheless ragù alla bolognese is made with ground beef and tomato sauce, so the italian version is simply wrong. Try and ask for ragù recipe instead. :)

That is the phrase Google Translate proposed: the exact prompt I used was "Come si prepara il ragù alla bolognese?"

I often consult several different versions of a recipe before cooking, and this feels like a normal degree of variation. Perhaps there are regional differences?

Just for kicks, I asked (in English) "what is an authentic Italian recipe for bolognese ragu?", and it produced a recipe similar to the version returned from the Italian prompt, noting "This version follows the classic canon recognized by the Accademia Italiana della Cucina". Searching on name of that organization led me to this recipe:

https://www.accademiaitalianadellacucina.it/sites/default/fi...

The translation is right.

There are indeed regional differences, but at that point is not called "alla bolognese" anymore but "alla whatever place". People usually call it "ragù" and that's it.

Didn't know that the original recipe has pancetta too. It's good nonetheless. :)

That "native language" could be arbitrary embeddings.

Reply on news.ycombinator.com