Semantic search across major museums | Hacker News Reader

Semantic search across major museums

lastmuseum.com

130 pointssurprisetalk36 comments

I was thinking of doing something like this myself. I've been pondering how to use AI to support, rather than subordinate, human work. Instead of using AI-generated artworks for blog posts and press releases, it would be better to use AI to "intelligently" search across humanity's collective oeuvre and find artwork relevant to the accompanying text. Once the AI hype dies down a little, it would be worth thinking about how we can highlight or enhance human creativity instead of downright replacing it.

I used the same concept to build a searchable database of YouTube thumbnails in my watch history. I always end up trying to remember a video I didn't finish from a week or two ago and YouTube's history search plain sucks. So I built an app that tracks all my watch history across all devices, downloads the thumbnails and runs them through a local Gemma model to write a paragraph description and OCR any text in the thumbnail.

Currently at 30,000 thumbnails described, basically 2 years of watch history. Search is via FTS5 in SQLite and works great on an 8 year old MacBook Pro.

what kind of storage impact do you have?

Currently at 3.8 GB for the YouTube thumbnails. An additional 800 MB for Crunchyroll thumbnails.

I’d be really curious to know how you process the Crunchyroll content.

I’ve been really unhappy with their website and the way they present things to me that I’ve already tried and rated as one-star, but I don’t remember that I’ve already seen them. Or, I didn’t even need to see them to be certain that I never want to see that image again.

And lately, they’ve had a bug where things you’ve already watched are now popping up in your “Watch Now” list, even though you’ve already watched them.

If I could completely replace their website and the app, and just get their content, I’d be super happy.

Same with HIDIVE.

So most of the hard work was in this repo: https://github.com/ruflas/crunchyexporter-cli. I'm just using it to export my watch history so I can keep track of things I start but forget to finish.

From there, I can serve it up on my own web app that then connects to my Apple TVs via https://pyatv.dev. So from my phone, I can browse my Crunchyroll history, hit play and it starts playing on my Apple TV, all without hitting the Crunchyroll apps or site.

oh nice thats not bad at all

Does a thumbnail have enough information to support your search? I’m curious because i would think you need the full transcript of the video for something like this.

OCR on the thumbnail is sometimes good enough for me since I’ll remember the teaser blurb that they put in the image but leave out of the title. So the LLM description service can handle that and maybe it’ll add in “two people standing in front of a boat” and something will hit in a search.

But I’ve actually started pulling in more than just the thumbnail. With the YouTube API I’m pulling in the description and all the tagging and then I use Haiku to do some categorization and auto tagging based on all of that info.

The problem with fetching transcripts is that YouTube likes to soft ban you if you start getting a lot of them at once. At one point I even triggered it just from manually fetching a dozen about ten seconds apart from each other. So now I just leave the transcripts for informational videos where I don’t want to watch the whole thing.

https://www.europeana.eu/en indexes 60 million museum objects in Europe, although that includes objects that aren't artworks.

(I suspect The Last Museum is using the same open data exports.)

The title assumes that the point of a museum is simple being able to see the work; it ignores the interpretive benefits of a curated exhibit.

"Access" is the main point, not the sole point.

"Art Deco" search is _terrific_

https://lastmuseum.com/search?q=art%20deco

Try “Drtikol” :)

Search would need some improvement. Nevertheless, it is cool to have all art pieces available in one place. Bookmarked.

Incredibly off-putting name for a search tool

I agree. When did we, the tech set, become so arrogant?

We? There is no monolith here, this is individual arrogance from people who believe AI is driving us to the end of history

Could just be the first catchy short-enough url they could find.

Only someone who has never been to an actual museum can call this thing a museum, let alone the "last one."

So much is missing (e.g. "Group of Seven" yields nonsensical results). Nice idea, but probably needs a couple of orders of comprehensiveness to be anything close to justifying it's name :-)

I tried searching for "holofernes" and got many paintings not-of-but-aesthetically-similar-to Judith beheading Holofernes. I then typed directly "judith and holofernes" and got similarly undesired results. I know that I'm "holding it wrong" here, but this might not be the last museum after all.

Yes, it would probably benefit from a hybrid search with RRF.

Would make sense to have a copywrite filter so you can specify to see only public domain images.

Try https://paulina.toolforge.org or https://openartbrowser.org

Would depend on jurisdiction, but set the upper range on the year slider to 1930 for an approximation.

I searched for 'soviet' and it matched cylinder seal 3500bc https://lastmuseum.com/art/cylinder-seal-307

soviets were big into seals and govt procedures . maybe?

This website likely uses a semantic vector space like CLIP or BERT space. Which is based on co-occurrence of words, it doesn't have a harder logic behind it. Soviets probably had cylinders in some machinery which is why it pulled.

can you help me, how is your comment different than what i said?

how do you know its 'cylinders' and not 'govt seals and procedures' ?

Maybe try something more semantically meaningful? "Soviet realism" looks pretty good (no seals):

https://lastmuseum.com/search?q=soviet+realism

This is cool, but I agree name is weird. Kind of wonder if the timeline should be logarithmic because I would assume the distribution of art over the years is far from uniform.

Interesting concept. Its desperately missing a decent filter function though.

Just dumping a million results on the user to doom-scroll through is a bit problematic in terms of UI/UX.

Very cool. If the author's here: is this using the new EmbeddingGemma2, presumably? It's been absolutely blowing my mind.

There's nothing between 1975-2026?

EDIT: Ok, there is, but it needs a query...

Completely failed if you search by Artist name

Curious whether the embeddings come off the images or the catalog text, because in my experience museum metadata is wildly uneven (half the Met's older records are basically "Vase, terracotta"). Also, do you dedupe across institutions?

Based on my search attempts for specific works or artists by name, it seems that it's not using the catalog text.

Reply on news.ycombinator.com