Million Song Dataset | Hacker News Reader

Million Song Dataset

millionsongdataset.com

I did my master's thesis (2017) using this dataset. I trained a neural network to predict musical features from the raw audio of the songs.

Unfortunately the 7digital ids were out-of-date, so in order to get access to the audio (30 second clips) I had to email another researcher who'd recently published work using the audio data and politely ask them to rsync me the audio XD

For anyone else who was curious about the song selection process:

<quote>

How did you choose the million tracks?

Choosing a million songs is surprisingly challenging. We followed these steps:

1. Getting the most 'familiar' artists according to The Echo Nest, then downloading as many songs as possible from each of them 2. Getting the 200 top terms from The Echo Nest, then using each term as a descriptor to find 100 artists, then downloading as many of their songs as possible 3. Getting the songs and artists from the CAL500 dataset 4. Getting 'extreme' songs from The Echo Nest search params, e.g. songs with highest energy, lowest energy, tempo, song hotttnesss, ... 5. A random walk along the similar artists links starting from the 100 most familiar artists

The number of songs was approximately 8950 after step 1), step 3) added around 15000 songs, and we add approx. 500000 songs before starting step 5. For more technical details, see "dataset creation" in the "code" tab.

</quote>[1]

What I really wanted to know was if it was a worldwide-music dataset or a more narrowly focused one. My guess based on the above is that it's mostly English-language music, mostly American - can someone who's worked with the data confirm/deny that?

[1] http://millionsongdataset.com/faq/#how-did-you-choose-millio...

I think this Dataset is probably unfortunately most famous for an incredibly flawed but very headlineable attempt, which you've almost certainly seen somewhere, by a group of researchers from AI and related fields (none of which had musical qualifications) to "objectively" determine if music has gotten worse over the decades. As usual, they arrived at the conclusion that it did by computing a vague number ("timbral diversity" and "harmonic complexity") for each song, and then showing a scary graph of that number going down over time.

Not sure what my point is exactly. I guess it's just another reminder of how easy it is to arrive at any conclusion you want using complex algorithms on big data.

For virtually any kind of art newer artworks will be on average worse than _surviving_ older artworks: the older artworks have undergone a selection process that weeded out the less popular ones.

This is IMO a more fundamental problem with any comparisons across long time intervals.

True for a naive approach, but I'd think looking at the chart toppers for, say, each week would give decently comparable data.

I actually did something similar! I wrote a small blog post about it here https://www.popnalysis.com/blog/lyrics-over-time/

But I came to the same conclusion as the op comment. Music can't really be judged by any one metric! But that doesn't mean that you don't gain insight into how music has evolved!

also, I did release my entire lyrics dataset that I scraped (about 500k) for free.

Yes, assuming you can find all the old chart toppers (and that this term has a consistent meaning over the whole time interval you're looking at). If, understandably, you have some gaps in old chart toppers, you need to assume something about them (probably that they were worse than all surviving ones, which helps if you're doing percentile-based comparisons).

Selection bias is also apparent in ranking sites for shows where seasons/sequels are ranked individually.

For example in anime, Gintama appears 8 times within the top 50: https://myanimelist.net/topanime.php

It's not because it's a popular show. It's just that people who didn't like the first few episodes have already stopped watching! And it's polarizing enough that the only people who stick around for so many seasons are the ones really love it. So it will get rated a 10 even for mediocre content.

I wonder if digital recording and processing has made music cleaner leading to less harmonic garbage.

Also... What if the poetry is better, and someone is singing acapella? How do you capture that?

Yeah, this is really the core issue, even within the music.

As a thought experiment, let's say someone invented a brilliant revolutionary new complex rhythm. Everyone went wild using it for a year. Then, once everyone knows it, artists start to mix it up by leaving out parts of it, leaving it implied, relying on listeners familiarity with the rhythm to make things work.

If you now tried to naively measure the amount of rhythmic complexity per song by counting percussion hits or similar, you'd see complexity take a nosedive. You'd also see people who missed out on the year complain about how bland the new rhythms are. But the songs actually got more complex. It's just that the complexity is only apparent to people familiar with the hypothetical revolutionary rhythm.

At the same time, people familiar with the new music will look back at the old music and be incredibly bored. They're used to finding enjoyment in the complexity of the implied rhythm, but there's just nothing there, it's all painfully spelled out and predictable.

I was so excited, until I realized the list hadn't been updated in 8 years... :|

Probably still good if you want to practice with machine learning on music.

I geolocated the artist list: https://geocode.xyz/874101666267029,share?export=GeoCluster

Next up, geoparse all song lyrics and compare those locations to the artists'.

That's cool, perhaps also link it from a centralized location like Kaggle? https://www.kaggle.com/datasets When I look for datasets, I first go to a place like that.

Found this dataset on Kaggle: https://www.kaggle.com/c/msdchallenge/data

Used this dataset in a university project to try and predict genre's from a number of features.

https://medium.com/modeling-music

In case this is not clear: the dataset does not include the raw audio of the songs, just extracted features.

Does this data set have a collection of chords in text? I'd love something like it.

The dataset is a bunch of SQLite files, so shouldn't be too tricky to interrogate.

You can get a subset (static.echonest.com/millionsongsubset_full.tar.gz), which is 1.8Gb compressed. The full dataset is 280Gb, and AFAICT, this does not contain the full audio.

There's a script on GitHub from like 8 years ago that apparently can get you the audio (but I would be super-impressed if that actually still works).

You can see a description of one song here: http://millionsongdataset.com/pages/example-track-descriptio...

i just got rick rolled by technical documentation

It looks like they have data about many of the individual notes in the song. I wonder if it could be possible to turn that data back into some sort of horrible midi version.

Reply on news.ycombinator.com