apwheele

I am currently using Perplexity for https://veruscite.com/. Verify citations actually exist -- so get in a list of 100, half I can check with crossref, but the other half I need to do a web search and fetch to make sure they exist.

So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.

I know what you mean by "cannot write out of a paper bag". Agree AI won't help those folks (I honestly don't know how anyone has enough patience to help them).

My above prompt does indeed work well for me, but agree would likely not catch those example errors you list.

I am maybe not using copy-editing the way you are (feel free to correct me what is more typical in the field). My prompt is not much different than "review the text for minor grammar and spelling errors, and list the corrections you would suggest". I accept the large majority of the suggestions.

For technical review of the content, I just send it out to other experts.

I am confused by this as well, shouldn't it be all white listed sites at the start?

Put in the todo list to check out. If you cut web search costs even further for flash/fast search (similar to Perplexity, which is $1 per 1000 for fast search) would make it more competitive (at least for my app!)

So I wish I could use Google for https://veruscite.com/, but the number of Google searches are a hard cap on the account! So yes that is fine for agentic coding, but for an app that relies on web-search is not sufficient.

I am currently using Perplexity fast search and fetch, and I am happy with that. I would try our Ceramic.ai, but I need to be able to fetch the pages as well (I do not want summaries).

If you check out https://veruscite.com/hallucinations there are much more banal examples than that article.

Part of the forensic work with this, I won't ever be able to tell the difference between "I as a human messed up" vs "LLM go brr and filled in everything". So I think editor discretion when to correct vs when to retract is fine. (I do think that PLoS One correction should have been retracted as well though.)

While this is true, it is still possible to take the probabilities and make determinations of recall and false positives using conformal sets, https://crimede-coder.com/blogposts/2026/ConfClassification

(Or just use a model to re-calibrate the probabilities, I like the conformal approach though as those rates are what I often care about.)

You just need some labelled data to generate the "corrected" probabilities (or thresholds to meet the specified error rates).

They are relevant and courts have considered the whole of a persons movements standard in terms of whether querying ALPR data constitutes a search, https://andrewpwheeler.com/2026/08/12/license-plate-reader-s...

Under current case law (Carpenter and recently affirmed in Chatrie) it will definitely be a search, IMO it is just when the sensors become dense enough according to the court (absent states do not make regulation themselves to require a warrant for historical data).

I sent it to the moderation email this morning. Will update if I hear back.

This is what the majority of mine look like as well. Just simple instructions for things I need to do repeatedly.

I have not even really needed a formal memory system. If I see an error happen more than once, I just say "hey add a note on this to agents.md". Tends to be verbose but overall works quite well for the projects I am doing.

Is there a good example someone can show of naturally generating real videos using extending and interpolation from the last frames? (For any of the video generators).

I mean I have not tinkered that much, but trying to even get a video to 30 seconds (I just want a cartoon AI avatar to narrate tutorials) has been incredibly difficult. They drift so easily.

Many AI videos you can tell just stitch short clips together. I just want a continuous scene for like 30 seconds to 2 minutes.

This is another riff on not embedding a full document, but doing a summarization of the document and embedding the summary for RAG. Nice usecase for high cardinality data!

I did not know about this when I wrote https://andrewpwheeler.com/2026/08/12/license-plate-reader-s... the day prior, but one of the things I did state was deleting data (except in the extreme case, like New Hampshire's 3 minute retention), will not prevent abuse. People just do illegal queries over and over again.

Part of the reason I wrote that post is it is better to retain the data indefinitely and have a warrant standard than continue to reduce the time period in which the data is retained.

This is already the current status quo on paper. I bet the majority of the cases IJ identified of mis-use the PDs already had this policy. (That is the same standard when you do a background criminal history check in New York, need to put in a case number, FYI.)

I did not want to go into too many technical details on the post -- I think you could do an AI audit at the point in time of the search query to prevent people putting in junk cases. That said I believe there will always need to be third party audits at a minimum.

Part of the issue is policies on paper are not effectively enforced. So people saying "just have policy X to prevent abuse" is a non-answer.

I wrote the post mainly because people arguing for limited data retention I think is bad for both sides -- it neither protects civil liberties and simultaneously makes it harder for long term criminal investigations.

I cannot read this article, but this may be partly confused what this function by Carbyne does. If there is a flood of critical systems, it just plays an auto message "If this is about X, thank you and we know about it" if you called from the same area.

I don't remember from prior articles if at that point you just stay on the line, or if you need to press a button. It is not going through an AI voice call tree.

My money is on they used agentic AI coding tools to help set up all the metrics/experiments with traditional models.

My experience like this was for a data scientist position at Facebook in 2022.

So the question was something like, here is a table of predicted probabilities for credit card transactions (hypothetical table, I do not remember the exact quantities), but it looked something like this:

    Amount, Probability, ActualFraud
       $5 ,    0.2     ,  0
     $100 ,    0.5     ,  0
       $3 ,    0.7     ,  1
       $1 ,    0.9     ,  1

And then the question was "what was the threshold used to maximize (precision|recall)?" (I cannot remember which metric they were asking for).

My response was that a proper decision model should take the dollar amount of the transaction into consideration, so the decision should more like be `$*p > threshold`, where this threshold meets your precision and recall for preventing dollars lost instead of the binary yes/no.

The interviewers English was not great, so after all that I was just like "well to answer your exact question, it could be anywhere between `0.5 < p <= 0.7`"

Did not make the second round!

An app to verify citations in peer review journals, https://veruscite-data.com

that is, to identify when papers have hallucinated references

can see an example of results at https://veruscite-data.com/share/3Pu3dwCfc-ixPgtHutVFXgMU7FY... for a preview. Also can sign up and get two free reviews.

It is in alpha (hoping to do Show HN in a few weeks), but for those interested I am working on an application to do this, https://veruscite-data.com/

Most of the folks on HN will be more familiar with genAI tools and can just use the skill Caleb and Isaac provided in this blog post. My tool is just likely more token efficient and has a GUI where you can review the extracted bib and edit it more easily.

So I agree with the general geist of this, a few counter-examples though:

If you have the raw log-probs, I show how to use conformal inference to set false-positive or recall rates, https://crimede-coder.com/blogposts/2026/ConfClassification

Some of the peer reviewed papers with the older models did show calibration was bad (not close to monotonic). This post with newer models (for one example, classifiying injuries) is not that bad, https://gmcirco.github.io/blog/posts/ai-calibration/calibrat....

I mean it just depends on the application, what level of error you can consider. But the second post shows how to recalibrate the scores as well if you need calibrated probabilities.

If you look at the raster image ChatGPT generated, that is fine. It is just this example (and other simple SVG icons I have asked for) result in pretty bad SVGs. It just makes me highly suspicious that the LLMs are learning shape primitives and extrapolating to new shapes, vs just having a big dictionary of prior examples and stitching them together.

So this is not my experience at all for asking about simple SVG icons for web-pages. Here is one of the examples I have tried for in the past, make a simple cartoon SVG knife for a map icon for a crime map.

https://x.com/CrimeDecoder/status/2080008114615537766

Can see the images for ChatGPT/Claude (Sonnet 5), and Gemini are all quite bad.

Jagged edge of LLMs. How do you explain being able to generate very complicated shapes in the Pelican example but cannot make a much simpler icon without just alluding to it is in the training data?

Not an open source, but I discuss it in my book with examples for OpenAI/Anthropic/Gemini, https://crimede-coder.com/blogposts/2026/LLMsForMortals.

All of the models, you need to have a consistent input to get the cache hit. So if you are chatting with a document, and change the system prompt, it will be a cache miss, even if the rest of the items are all the same. If you even pass in the document in not the same order as the prompts, it will be a cache miss. Or if you add tool calls or structured outputs, it will be a cache miss. (Since those generally go at the beginning of the prompt call, not at the end.)

Most of the time when reading documents from URLs directly it will never cache. (Need to typically pass in the bytes directly, or use the provider document store index.)

Gemini has a 4096 minimum token size with the 3 version models before even getting a cache hit. OpenAI it is lower (1024), and is automatic, but only happens in increments of 124. Anthropic can also get cache hits at 1024 tokens, but you need to explicit ask for it (and pay extra).

Caching by default typically lives for 5 minutes since the last cache hit across providers. But some of them you can ask for longer. AWS for Anthropic models can be tricky with multiple endpoint routing, so can get cache misses if it happens to route to a different endpoint.

Just as even a counterpoint to this, I have asked the LLMs to attempt to generate SVG icons for websites. Even though I have requested things much simpler than a Pelican, they have all tended to do quite poorly in my examples.

Because of this, I presume the Pelican has been in the training data for at least a year+.

The models are very useful, I am afraid they have fundamental limitations though generalizing (it is just hard to evaluate effectively). So it will just be whack-a-mole "can your model do X", and there will always be a new X.

I think this is a good meta-lesson for Kaggle. When you have objective metrics to hill-climb towards, AI can do quite well. When you just phone it in and rely on LLM as a Judge, the results are not so great.

Agree with the web search point. (I would like for Perplexity to start to offer more models out of the box integrated, like they do now with OpenAI/Gemini models.)

Processing 500 retrieved chunks here (probably with an additional LLM) will also add a bit of latency (not shown in these graphs!).

So for user facing apps, that scenario is probably not feasible (more like filter 10 chunks). Which as the parent of this comment suggests is fine to add in extra context given the current size of context windows.

If I may shamelessly suggest my own book for those looking for a more basic intro to calling the APIs, including a chapter devoted to agentic patterns across the different SDKs (openai, claude, and google), https://crimede-coder.com/blogposts/2026/LLMsForMortals.

Basically I wanted to write a book that did not spend the majority of time on training LLM and architecture, as it is just not relevant to the majority of software engineers (either using agent coding tools to help write code, or using the LLM APIs as backend parts of apps you are building).

This Hitchhikers book has many examples of API use as well, but is very "here is a wall of code". Mine is a bit more gentle progressively building up simple examples -- e.g. I show how to write python code to call a tool in a loop. Then show how you can feed back the error and have the LLM update. Then show how you can do that without manually looping through the agents SDK.

Off topic, but Simon's Pelican on a bike SVG makes me think they are using that is the current models. I ask for much simpler SVG icons periodically (to insert into webpages, map icons, chart icons, etc.) and all of the models tend to do poorly in my experience.

Most recent example was a V to look like a checkmark. Seemingly much simpler than a pelican riding a bike, but Claude/ChatGPT/Google all did quite poorly.