Comment by exogen | Hacker News Reader

Comment by exogen | Hacker News Reader

I missed the original HN and twitter threads referenced in the post, so I might just be repeating something that was already said there...

But, in nearly all cases I would trust a bespoke Wikipedia scraper over using the output of Wikidata or DBpedia. Not to disparage either project, because they're great ideas and good efforts. I have a firm grasp of RDF and SPARQL queries (used to work with them professionally), which also makes them tempting to use.

One issue is that Wikidata tends to only report facts whose subjects or objects themselves have articles (and thus Wikidata entities).

For example, compare the "Track listing" section of Carly Rae Jepsen's Curiosity EP on Wikipedia vs. the "track listing" property on Wikidata.

Wikipedia has:

    1. Call Me Maybe (link)
    2. Curiosity (link)
    3. Picture
    4. Talk to Me
    5. Just a Step Away
    6. Both Sides Now (link)

while Wikidata has:

    1. Call Me Maybe
    2. Curiosity

So not only has it ignored any tracks that aren't deserving of their own articles, but it also missed one that actually does have an article (track 6, a cover of "Both Sides, Now").

> Others asked about the data quality of Wikidata, and complained about the huge amount of bad data, duplicates, and the bad ontology in Wikidata (as if Wikipedia wouldn’t have these problems. I mean how do you figure out what a Wikipedia article is about? How do you get a list of all bridges or events from Wikipedia?)

Often the problem isn't that Wikipedia is wrong, it's that Wikidata's own parser (however it works) doesn't account for the many ways people format things on Wikipedia. With a bespoke parser, you can improve it over time as you encounter edge cases. With Wikidata, you can't really fix anything... the data is already extracted (right or wrong) and all the original context lost.

Scientific Articles are in a similar situation: when importing one from a bibliography database, you won't always find every author... So people made an alternative prop "author name" and some disambiguation tools that allow users to gradually replace those with "author" links to real persons.

Let's put aside the question whether each song ever written should be in WD: I believe all of this data, modeled more elaborately, is available on MusicBrainz. There's a difference between a work (eg "Both Sides Now") and its particular rendition in an album (as you said that track is "a cover"), and MusicBrainz makes that distinction and captures both, but I think WD doesn't (I don't work on music in WD, so I haven't checked).

If you really want all this data in WD then I guess you could import it from MusicBrainz... a massive undertaking.

> Wikidata's own parser (however it works)

There's no such thing (in contrast, DBpedia has the dbpedia extraction framework, which is fairly good but not perfect and suffers greatly from the various ways people use to describe the same thing). WD has tools like QS and wikibase-cli, and people write bots to scrape and contribute specific kinds of data.

BTW the Wikipedia link https://en.wikipedia.org/wiki/Both_Sides,_Now#Other_recordin... is wrong (invalid): you can't have two anchors in a link.

Cheers!

acute observation! there is something here to be teased out .. about.. the final product is a human readable page all these years, and that human readable page got better in adhoc ways and most all of those improvments stuck..

compare to the RDF efforts, who ride a rigorous math-y perspective and with a far, far smaller development crowd right away..

> So not only has it ignored any tracks that aren't deserving of their own articles, but it also missed one that actually does have an article (track 6, a cover of "Both Sides, Now").

In other words, "scraping wikipedia" is the answer to the question implied in the HN title to this post. :)

I'd suggest that in this case one should consider using MusicBrainz, in order to get more comprehensive and better results than either with Wikidata or Wikipedia.

I wouldn't say the data is better, just different. Instead of "how do I extract the info I want?" your problem becomes too much data to sift through. See my comment here: https://news.ycombinator.com/item?id=24992600

That obscures my point.

My point is that even if "Wikipedia" may not be the best tool, "Wikidata" is the wrong tool because the data is wrong.

Reply on news.ycombinator.com