Hacker News Reader

Comment by inflatableDodo | Hacker News Reader

I wonder how many people are doing this now and what media they are storing to.

Reddit's /r/datahoarder is full of people archiving all sorts of things that I wouldn't expect to be archived. Recently there was a community effort to archive YouTube's annotations before they were removed. Link here:

https://old.reddit.com/r/DataHoarder/comments/b7imx9/youtube...

Internet Archive has been recording TV news since 2001:

https://archive.org/details/tvnews

They have projects based on this as well. One of the prominent ones was the Sep 11 Archive. https://archive.org/details/911

Kind of tangentially related but this compilation of TV footage in the morning before the 9/11 attacks [0] is always slightly unsettling to me.

[0] https://www.youtube.com/watch?v=IMVTB2aVUg0

I only watch TV through downloads (get_iplayer & youtube-dl), and I have kept it all [edit: only things I downloaded intending to watch, not all TV]. This is since about 2008:

    fs:/volume1/media   14T  8.0T  5.6T  59% /mnt/media

The reason? Why not - disks are cheap. The hard thing is deleting something since I don't know if I might want to watch it again. And at almost no real cost I don't have to make that decision.

Edit: Some stats: 20,269 TV programmes; 11,830 radio programmes; 925 movies

I wish I started doing similar thing way earlier, because now I can't refer to a lot of content that vanished from the net for a number of reasons (creators getting bored, flagged out from youtube by bots that random haters set up, bogus copyright claims from impostors that upload all sorts of stolen things into contentid in order to create a quick buck, UMG taking down videos from music theory channels despite it obviously falling under the Fair Use and so on)

For music on youtube, this trivial ytdl wrapper is essential:

    $ cat bin/youtube-mp3 
    #!/bin/bash -
    exec $HOME/d/youtube-dl/youtube-dl -t -x --audio-format=mp3 "$@"

Saved to my Squeezebox library, it's available to play forever.

Doesn't the audio quality take a hit when you transcode to MP3 though?

Possibly, but on the flip side neither my car nor my first-gen SLIMP3 can play other formats natively.

YouTube audio is usually aac. Sometimes I have saved audio as an .aac file without re-encoding. (Not using the same scripts, what I have done typically is get the mp4 and do: ffmpeg -i foo.mp4 -acodec copy foo.aac)

Any audio player worth its salt should play .aac files. I think for the most common types [low-complexity] the patents are expired. (Curious if somebody has one that doesn't.)

Probably, but it's already down to ~150kb by the time you download it.

Do you maintain any kind of database or additional metadata alongside all of those video files? I only ask because the default filenames generated by youtube-dl don't give you anything more than the title and ID for a particular video and I imagine it's difficult to search among tens of thousands of media files without contextual data about the files.

No, just filenames and whatever metadata is stored inside the file. For get_iplayer this isn't very much unless you have "AtomicParsely" installed too, but I uninstalled AP after it segfaulted once too often. Note for BBC TV programmes you can search on the "PID" (which is part of the filename), eg: try searching for "m000427z".

However I also download subtitles where available so I guess I could search those as they are just text.

yt-dl has a full-fledged output template system. It's just the default output template that is only the title+id.

https://github.com/ytdl-org/youtube-dl/blob/master/README.md...

Also, it has the --write-* switches, in particular --write-info-json, which allow to store metadata data. in files, and --add-metadata to embed (some) of the metadata in the file itself.

Of course, the metadata is only as good as the site in question provides + the yt-dl plugin handing a particular site supports.

I was familiar with the -o template system for filenames, but wasn't aware of the --write-info-json flag which is very useful to me. It makes a lot more sense than overloading the filenames with usernames and everything else only to parse it out after download.

Nice man!

Would you consider donating a copy of the TV news programs and radio to the internet archive?

Stephen Wolfram (the man behind Mathematica) had this odd but fascinating blog entry:

https://blog.stephenwolfram.com/2012/03/the-personal-analyti...

He captured every keystroke (over 100 million) on his computer.

This is a great display of Mathematica's power in data science and plotting. Somehow I'm feeling compelled to reach for my wallet....

> Somehow I'm feeling compelled to reach for my wallet....

Which is a great display of Mathematica's power over people's wallets

Oh. If I didn't know any better I would suspect Stephen Wolfram made a blogpost for shamelessly advertising the data science capabilities of Mathematica for the purpose of selling it to data scientists.

I think the blogposts are being generated by a collection of cellular automata that are being created by a learning algorithm with access to that dataset. In other words, Stephen Wolfram.

I happen to have been doing this since Feb 10 this year, as I've been using a USB DVB card to capture/watch the news. Up to 370 GiB so far, just storing the MPEG-TS streams on hard drives.

Out of curiosity, why?

Why do I watch the news? I don't like the idea of relying on the news sites here (New Zealand) in getting a brief overview of the current stories.

If you meant why I'm keeping them .. I just don't really like deleting things. If the space usage ends up being impractical, I'll probably just end up reencoding them.

Edit: to clarify, by "the news", I mean the hour-long show (though I skip the sports segments) that's on at 18:00 every day; we don't have the "24-hour" news cycle that seems to be common in the US.

Thanks!

Would you consider donating a copy of the TV news programs Internet Archive?

When I was living in Australia (2007-2012) I had a tv card in my computer so I could watch tv in my room, and I could record another channel while watching. So I used to record shows to mpg to watch later.

I would bet there are many people with very extensive porn collections. The more specialized, the more likely they would create a library.

I remember the first time I was in awe of storage capabilities was roughly 2003 when someone sent me a screenshot of his pirated porn collection, it was in the hundreds of TBs, at that point already, mostly stored on Disc spindles. I often wonder if he's stopped accumulating at some point.

In /r/datahoarders people are abusing Google Drive Quotas for the same purpose, but the capital required doesn't remotely compare to what it seems some people are willing to invest.

I think private torrent trackers are your best bet when looking for this kind of obscure content.

I think / hope those are archived as well; e.g. what.cd and waffles had a huge archive of all kinds of rare music, bootlegs, obscure bands, limited releases, etcetera. I hope there's an organization that made sure those were archived.

Are museums exempt from copyright laws for archival purposes?

A number of sites filled the void left in place of What.cd and preserve even more content than before. The community is very dedicated.

Reply on news.ycombinator.com