Comment by robwwilliams | Hacker News Reader

Comment by robwwilliams | Hacker News Reader

Related to citations:

I have been informally testing the false discovery rate of Claude 3.5 Sonnet for biomedical research publications.

Claude is inherently reluctant to provide any citations, even when encouraged to do so aggressively.

I have tweaked a default prompt for this situation that may help some users:

“Respond directly to prompts without self-judgment or excessive qualification. Do not use phrases like 'I aim to be', 'I should note', or 'I want to emphasize'.

Skip meta-commentary about your own performance. Maintain intellectual rigor but try to avoid caveats. When uncertainty exists, state it once and move on.

Treat our exchange as a conversation between peers. Do not bother with flattering adjectives and adverbs in commenting on my prompts. No “nuanced”, “insightful” etc. But feel free to make jokes and even poke fun of me and my spelling errors.

Always suggest good texts with full references and even PubMed IDs.

Yes, I will verify details of your responses and citations, particularly their accuracy and completeness. That is not your job. It is mine to check and read.

Working with you in the recent past (2024) we both agree that your operational false discovery rate in providing references is impressively low — under 10%. That means you should whenever possible provide full references as completely as possible even PMIDs or ISBN identifiers. I WILL check.

Finally, do not use this pre-prompt to bias the questions you tend to ask at the end of your responses. Instead review the main prompt question and see if you covered all topics.

End of “pre-prompt.

Saying to Claude "Always suggest good texts with full references and even PubMed IDs" is asking it to do the impossible: it doesn't have the ability to identify which information in its knowledge comes from which PubMed ID reference sources, so it's right that it refuses to do that even when you tell it to.

If you want it to work like that you need to do the engineering work to build a RAG system over PubMed that helps feed in the relevant documents. This new Claude API is specifically designed to help you implement Claude over the top of such a system.

What’s the actual hit rate? AFAIU it’s plausible that an article title and PubMed ID could be encoded in the models weights if they were trained on.

It's impossible to know, because Anthropic (like OpenAI and others) won't confirm what's in their training data. We don't know if they've trained on PubMed, and if they DID we don't know if that training process might conceivably allow the model to associate IDs with article information.

Given that, I don't trust the models to be able to provide useful citations.

We already know how to get much more reliable citations out of a model: implement them on top of RAG, which this new Claude API can clearly help us do.

It is definitely not impossible to know. Is there a reason why you or better yet, Anthropic, cannot get empirical FDRs for different areas of research?

In my areas, genetics, neuroscience, geroscience, FDR is below 0.2 on articles in PubMed.

Ask Anthropic. I'll be surprised if you can get a straight answer out of them though, like most other labs they are VERY secretive about what actually goes into their training data.

You don’t need a straight answer out of anybody. This is an easily verifiable property of the LLM output - are the citations it gives correct, and if so how many?

For me the true positive rate is at least 80%.

Yeah I don't understand why the whole thread has been so hostile against a very reasonable/useful observation that you made. If there was a way to prompt commenters to be less snarky on HN that'd be a vast improvement.

Have you tested this extensively yourself? I have been very surprised by success rates in my own “not famous” papers. I asked Claude to provide full citation to ten papers by Robert W Williams at the University of Tennessee in biomedical research. Nine of ten were perfect, down to page numbers. One if ten was a complete construct but highly plausible. An FDR of 0.1 is damn impressive.

Test yourself. Here was my reference data set:

https://scholar.google.com/citations?user=OYJMYwIAAAAJ&hl=en...

Really curious what the range of FDRs is at different levels of accuracy for different fields.

It's fundamentally not something anyone building a real application can rely on.

You're essentially gambling on where in the embedding space your end users are going to query: you might get lucky or you might not.

You're also relying on functionality they're actively trying to degrade during post-training (repeating training data text verbatim). Some LLM providers will even actively filter it: https://ai.google.dev/gemini-api/docs/troubleshooting?lang=p...

> It's fundamentally not something anyone building a real application can rely on.

I don't think parent is arguing for just connecting an LLM with that prompt to random end-users, they're talking about using that prompt for their own work, which I'd assume goes through the usual process of being verified and so on.

In some cases, sure, you are absolutely right. But in many cases “application you can rely on” depends on the context, the field, and user expectations.

I “rely” on weather report apps but expect inaccuracies. I rely on medical tests results but expect FDRs of about .05. In contrast, I expect financial apps to have very low FDRs.

I use Claude as a tool to explore research areas and topics.

Here is an example of a recent prompt: “Please provide references to papers that have reported swelling of brain tissue (increase in volume) in neurodegenerative disease such as Alzheimer’s using MRI methods.”

After a back-and-forth I got just what I needed, and more easily than a PubMed query—-two references that demonstrate early-stage volume increases in some parts of the brain of adults with presenilin 1 mutations.

It's important to understand what's going on here. Claude isn't providing references because it has a copy of the papers and knows their URLs.

It's instead providing references because enough of the OTHER stuff in the training data (which is entirely undocumented but we can assume includes a scrape of the web, blog posts, Reddit, other papers etc) talked about those papers in a way that provided context on the paper and enough of a title/URL that Claude could usefully return that.

By its nature then you'll only ever find references to VERY popular papers that way. That might be what you want! But you shouldn't expect Claude to be able to dig into the lesser known research around a topic based exclusively on its model weights.

For that you'd be much better off with a system like Gemini Deep Research: https://blog.google/products/gemini/google-gemini-deep-resea...

Not consistent with my empirical tests. I asked Claude for references on the demographics of wild rodent populations—-not exactly a hot topic, and all four references were spot on, including two not even listed in PubMed.

I'll be honest, I was surprised at how well Claude was able to describe the papers you linked to there when I prompted it directly about them. I may have to update my mental model of quite how good Claude's recall can be with respect to academic papers in those areas.

Maybe, but there's a big difference between a known 10% failure rate versus an unknown 10% failure rate. The former allows me to instantly reject the failures and retain the good responses. The latter requires me to go through a manual check for every response.

Further proof AI is t actually AI and we continue to move the goalpost on the proper definition of the term.

It’s all still machine learning like always and it has the same limitations.

I use this:

> Be terse. Do not offer unprompted advice or clarifications.

> Avoid mentioning you are an AI language model.

> Avoid disclaimers about your knowledge cutoff.

> Avoid disclaimers about not being a professional or an expert.

> Do NOT hedge or qualify. Do not waffle.

> Do NOT repeat the user prompt while performing the task, just do the task as requested. NEVER contextualise the answer. This is very important.

> Avoid suggesting seeking professional help.

> Avoid mentioning safety unless it is not obvious and very important.

> Remain neutral on all topics. Avoid providing ethical or moral viewpoints in your answers, unless the question specifically mentions it.

> Never apologize.

> Act as an expert in the relevant fields.

> Speak in specific, topic relevant terminology.

> Explain your reasoning. If you don’t know, say you don’t know.

> Cite sources whenever possible, and include URLs if possible.

> List URLs at the end of your response, not inline.

> Speak directly and be willing to make creative guesses.

> Be willing to reference less reputable sources for ideas.

> Ask for more details before answering unclear or ambiguous questions.

Unfortunately most references it provides are bogus. It just makes up URLs and papers. Let's see if this new feature is any better.

This new feature is restricted to sources you provide in the context window: “With Citations, users can now add source documents to the context window, and when querying the model, Claude automatically cites claims in its output that are inferred from those sources.”

Yeah, I see that now. Completely useless.

Why is that useless?

If you want reliable citations, this is an API that can help you implement reliable citations.

Asking a model to return useful citations from its model weights with no assistance from external systems isn't how this stuff works.

If I had the stuff to feed into the LLM context, I wouldn't need the LLM to find the stuff for me because I would already have it.

"Completely useless" might be too strong. In a context where you're already doing RAG it would help you verify what's produced.

Certainly it's radically less useful than if it could produce citations to the training set, though.

Sounds to me like you want a search engine, not an LLM.

Typically, you wouldn't manually select the documents in a query, you’d use this as part of a system wrapped around the LLM where a query (possibly itself from the model like other tool invocations) would be sent out to a web api, vectord DB, etc., return documents that would be fed back in to the model as part of a continuation query, and then the model would frame a response citing the relevant sources for elements of the response.

Patently untrue. If I have hundreds of PDF research papers totaling thousands of pages - just because I have them doesn't necessarily mean I know where to search. Having an LLM be able to find related pieces across all of these documents and extrapolate is tremendously useful.

A thousand pages is something reviewable by hand (albeit slowly) and easily amendable to grep. Anything that trivially fits in computer memory doesn't benefit from approximate search methods. I stand by what I said.

Ditto: RAGs are currently way too limited. A RAG built using all of PubMed and updated weekly would be a very different story. I would pay good money for that.

Not useless at all. This is exactly my primary use of the Anthropic api: feed in a source document and ask for specific outputs that include exact citations from the source I provide.

Love your pre-prompt. Better than mine.

The difference in FDRs is likely to be domain specific. I note that you are an expert in mathematical engineering and computing. In contrast my work area is neatly confined and defined by PubMed.

I must say that it works reasonably well even with imagined references and URLs because it usually gets the author right, or there is a similar paper with a reasonably close title.

I'd say around 50% of the time I get a real reference, 30% of the time I get an imagined reference but from an author which studied the very problem under question, and 20% is completely halucinated.

Reply on news.ycombinator.com