Hacker News Reader

Comment by toomuchtodo | Hacker News Reader

I asked for a data dump via email, waiting to hear back. Ripping it all out is going to cost someone a pretty penny since it's hosted in AWS, better to ask someone to fill and ship some SATA drives. Then off to the Internet Archive!

View on HN

I saw a companion AWS blog on this: https://aws.amazon.com/blogs/publicsector/smithsonian-3-mill...

Looks like its part of the public dataset program so you can probably just ask for the bucket name and get full free access to everything.

I can confirm that this dataset is indeed part of the AWS Public Dataset Program. We were finalizing some details, but this dataset is now listed in the Registry of Open Data:

https://registry.opendata.aws/smithsonian-open-access/

Enjoy! I’m personally very excited about this dataset, and couldn’t be more impressed with the people, mission, and, well, everything at the Smithsonian.

Source: I work on the AWS Open Data team / had a small role in this (normal lawyerspeak caveats apply that my views and opinions are my own)

Thank you!!

This is weird. The Smithsonian page [1] says "Data hosting provided by AWS Public Dataset Program", and there's the Amazon blog post you linked, but the data set seems to be missing from the registry Amazon publishes. [2] I guess no one submitted a pull request yet? [3]

[1] https://www.si.edu/OpenAccess

[2] https://registry.opendata.aws

[3] https://github.com/awslabs/open-data-registry

> just ask for the bucket name and get full free access to everything

Jed or jeffbarr, what say ye?

Disregard! Pete’s comment is sufficient for bucket enumeration.

How much data do you think it encompasses? Approaching/surpassing Petabytes?

Couldn't even hazard a guess, I can send a Backblaze-like chassis if needed, but if there's no cost to pull from the AWS public S3 bucket, I'll just spin up some VMs and enumerate everything into items in the Internet Archive. Trying to find the balance between time, cost, and inconvenience for all parties involved.

still wondering/trying to get a sense of how much data has been accumulated in the open data digitization process. Gleaning that it might be multiple petabytes based on the open data sessions that were going on last few years but dunno...

Some of the ppl interested in the data and a lead at AWS got back to me on it:

aws query on the open dataset comes back with:

Total Objects: 4649789

Total Size: 312.5 TiB!

Reply on news.ycombinator.com