Comment by jsnell | Hacker News Reader

Comment by jsnell | Hacker News Reader

I tried [when is ambulance coming out], and you did correctly understand it was referring to the movie. But the result is strictly worse than the same search on Google (which also knew it was a movie, but actually gave an instant answer to the question rather just a IMDB link, and answered the question correctly for my location unlike the IMDB page). Bing also returned an instant answer, but their answer is even more wrong (the release was pushed back by a couple of months, Bing claims that the movie would be released at the future date of February 2022).

Then I tried [will there be a wheat shortage this year?], and the top result was a lunatic fringe prepper blog, from which you extracted a confident answer that there will definitely be a wheat shortage. Google doesn't try to answer the question (which I'd argue is correct for something this speculative), and only links to reputable news sources all of which look relevant. Bing is the same, but mixes in some more alternative results at the bottom of the page.

Then, since you said that this worked best in a conversational manner I asked [is it due to the war?], obviously referring to the previous question. Rather than keep the context of that previous search in mind, you replied with information on the current local weather. (In Fahrenheit, for a location in Europe).

At this point, combined with how incredibly slow the searches are, my goodwill would be gone as a real user.

Thank you for trying it out, and giving us feedback on the searches. We don't log searches, and can't see what results users get, so we rely heavily on community feedback to improve things when searches go wrong.

Interestingly, with the Ambulance search I see:

```I found this deep answer on imdb.com:

Laurits Munch-Petersen (based on the film "Ambulancen" written by) Lars Andreas Pedersen (based on the film "Ambulancen" written by) Stars Jake Gyllenhaal Yahya Abdul-Mateen II Eiza González See production, box office & company info Coming soon Releases April 8, 2022 40 User reviews 28 Critic reviews Videos 3 Trailer 2:58 Official Trailer ```

I'n not familiar with the movie, but one thing that really helps us improve results is feedback on what the right result *should* be to help retrain based on examples for ranking.

At the moment, the alpha is very US-centric (imperial units only, no local business/maps etc), although our early user community is global and they're pushing us to add better international support quickly. Funnily enough, it's often decent at searches in different languages as it uses APIs and live content sources. But the front-end is not localized. As a small two person team, we hope to get some help on this front in future. There is a meaningful subset of folks who prefer not to have locally filtered results. So we have some work to do figuring out the right approach there.

With the wheat result, that's an interesting example. The other results look reasonable but the ranking was out there definitely. The Q&A process picked the wrong representation of the consensus result from across the set of similar views from Reuters, Fox etc.

This is a good case for user control of results I think.

We're working on anaphor resolution and better disambiguation but have a lot of work to do there. There are some simple things already in the alpha. eg try putting in "Paul Graham" :)

> Interestingly, with the Ambulance search I see:

Right, that's a legit page info for that movie. But your search engine doesn't actually extract the answer to my question ("releases april 8th") from the page, and this should have been about as easy as it gets. The answer happens to be in the snippet, but the snippet is unreadable.

Now, I wasn't seriously expecting you to have localization of results at this point, but the query wasn't meant to trip the system up either. I only realized this wasn't a globally synchronized release date when comparing the results. (And again, this was about the easiest possible case for localization. IMDB has a page with structured data for release dates by country, and you clearly are using geolocation for the query that produced local weather results.)

Yes any localization is very much a future feature. There are some small functions that use it that are things our user community has asked for, but we're very much taking an iterative approach. I think with a small team you have to do that, and accept that early on you can't do everything but that you can keep iterating and improving quickly based on the things that users say are most important.

The most important feature for our early users so far, based on the feedback we get, is factual information question answering, and there are some classes of query where Andi is already very useful. And as it gets better we hope it can save people a lot of time. Reader view is popular for the same reason because it saves people time and distraction.

At the same time, I think everyone (myself included) likes to play "let's trip up the AI" with something new like this, and it's fun and a great way to get training data and share a laugh on our Discord. When we don't log searches, it's some of the most valuable data we can get :)

A friend mentioned to me that you're a software engineer at Google.

Thank you for taking the time to have a look at what we're building with Andi, and giving us detailed feedback on the results. I mentioned that we don't log searches, so examples like the ones you found where things go wrong are very helpful for improving results. And that's even more the case when the examples are from folks who work at Google. In general, Googlers are more likely than most people to be able to find bad results.

Long term, we think our approach can do better than Google's, even though we have much work to do. As an illustration of the difference of approach, I thought you might be interested in this example where Andi gets the right answer, and Google gets it completely wrong, because it is trying to extract snippets and re-format them.

1. "what is the gdp per capita of china vs new zealand"

Google displays a table that shows completely wrong information:

``` GDP per capita [+] 2020 €36,371 GDP per capita [+] 2020 $41,165 ```

Andi's answer:

"In 2020, the gross domestic product of China was about 10500 US dollars per year per person. In 2020, the gross domestic product of New Zealand was about 41478 US dollars per year per person".

(sourced from Wolfram|Alpha in this case)

I just tried 'when will the movie ambulance be released?'. Answer was 'The movie ambulance will be released on April 8, 2022'.

Yes, with the semantic-style search Andi does best finding answers when it has specific questions with enough detail. That will increase the confidence that the result is factually correct.

Better questions = much better answers

So specifics help you get better results.

Reply on news.ycombinator.com