> Your domain is weird/rare, so LLMs are terrible because their training data is very limited.
[1] https://arxiv.org/abs/2404.03502
I am not a big fan of LLMs so I try them once in a while, asking to "implement blocked clause decomposition in Haskell." They can recite a paper (and several other papers, with references) almost verbatim, but they do not posess enough comprehension of what is going on in these papers. As time passes by, with each new LLM, the level of comprehension drops. Last time LLM tried to persuade me to write code instead of providing me with code.
With this example usage:
-- Example usage:
let clause1 = Set.fromList [1, 2] -- represents (x1 ∨ x2)
let clause2 = Set.fromList [-1, 3] -- represents (¬x1 ∨ x3)
let formula = Set.fromList [clause1, clause2]
-- Decompose the formula
let (nonBlocked, blocked) = decompose formula
Did it do just from the prompt or you had to nudge it? Can you share full chat history?
Can you share actual interaction on their site?
https://support.anthropic.com/en/articles/9450526-how-can-i-...
> It’s verbatim, I promise.
The training data is always limited, because if there was an existing software that already did what I'm doing, I'd be using it instead of writing it :)
So by definition, unless I'm learning how to program and doing exercises that thousands have done before me, I'm doing something for which there is no training data.
> The rest of us peasants tends to recombine previously solved problems into to novel solutions to solve business needs, and that works rather well with LLMs.