AWS Data Exchange
aws.amazon.com
aws.amazon.com
Incidentally, what is the open source alternative of this? Data is so cheap that it should be actually free, unlike counterfeit nike shoes.
(Does a bittorrent tracker specifically for research data exist? Edit: there's http://academictorrents.com/)
Getting data is not cheap, and maintaining a dataset is certainly not cheep either.
There are many public-domain datasets:
* https://aws.amazon.com/opendata/public-datasets/
* https://cloud.google.com/public-datasets/
* https://github.com/awesomedata/awesome-public-datasets
> Data is so cheap that it should be actually free
I feel like you haven't had the opportunity to work with data that has value.
Data vendors sell valuable data all the time to essentially anyone who is willing to pay their asking price. I don't see how this is any different? Just because aws is trying to run a marketplace doesn't suddenly make the data "public" in an open/free sense, there's still a price tag attached to it.
> and, like nike, won't be selling itself on amazon
Not every data vendor (or retailer since you keep bringing up Nike) has the necessary brand recognition, marketing budget, or technical proficiency to only sell direct to customers: that's why centralized marketplaces and alternate distribution channels exist.
if it s truly valuabe it isn't sold to 'essentially anyone'. like high value financial info or security.
That's how data companies make money, by doing that work for you.
I feel like you haven't had the opportunity to work with data that has value.
It will be a very busy, but lucrative time for corporate law firms as they battle things out.
Data that has an useful business or predictive purpose that is clean and constantly updated is not cheap at all.
This really depends on the data. In pharma the right 50 bytes of data can be worth billions. Not all data is personal product preferences for add targeting.
[1] https://aws.amazon.com/marketplace/pp/prodview-bwf7mapyyjzom...
You shouldn't allow people to sign up to "autorenew" at the end of their free trial period without clearly displaying the price that will apply when the trial ends.
We’ve built this index for autonomous driving datasets (https://scale.com/open-datasets) and are building that out for other domains right now.
Open source data has been a pillar to progress in ML (starting with ImageNet). It should continue to be the case that data that enables researches is sufficiently democratized.
AWS charges are really obscure. I think they make an effort at making it hard to estimate/determine how much you'll be charged. (ie: You charged for EC2 time, bandwidth, storage, etc...). That makes it very hard to see where the money is going.
I’m building out an app and we receive documents from all kinds of vendors from all over the country. The app is for our business to manage our client data. I was hoping to find a list of business I could throw in the db rather then piecemeal add the addresses in one by one as the documents come in.
I looked at some of the data service providers (infoUsa I think was one and d&b being another), but one dataset for just business names and addresses they were asking $50,000 for. I think my use-case is unique in that these companies typically sell this data as sales lead data which it definitely is not in my case (we don’t even sell b2b).
Anyone know of anything like this? I suppose I could just scrape phone books but I think if I can’t find the data we will just resort to one by one entry.
Very interesting idea - many thanks!
https://www.wired.com/2007/12/why-anonymous-data-sometimes-i...
I dug through it just out of morbid curiosity and unfortunately stumbled upon a few searches indicating that a friend privately suffered a miscarriage. This was only possible because I recognized her as the only person that I knew at the intersection of a few other search terms that were correlated to the same 'anonymized' id.
GP is assuming a naive starting point when that is rarely the case.
I have an open source project on crawling public datasets and make them searchable in one place: https://github.com/findopendata/findopendata.
What could go wrong for information providers where Amazon controls their market and infrastructure? They become commoditized "data providers". They are coerced into profit sharing with Amazon. They are eventually replaced by Amazon-provided data.
I won't buy from this market because I see where this is heading. I use the same reason that I apply for not buying many other services and products from Amazon. It offers no additional value other than minor convenience to a customer at a much greater cost to the economy and providers.
Buying local isn't just for produce.
If your company goal is to sell commodity data and make money in volume rather than high margin less frequent sales then maybe this is not so bad.
To your point about Amazon eventually taking this over, I do believe that to be the case when there is near ubiquitous demand for the data type. And just looking over what is in there now, these are some pretty specific datasets.
I'm not saying you're wrong, I will be considering your comments for quite a while.
I have said this on here before and I think it still holds true. Amazon is like Jay Leno.
Conan O'Brien once said:
“Hosting The Tonight Show has been the fulfillment of a lifelong dream to me. And I want to say to the kids out there watching, you can do anything you want in life unless Jay Leno wants to do it, too.” Apr 17, 2014
https://adexchanger.com/the-sell-sider/third-party-data-is-a...
1) It's a continually updated file, so if you didn't subscribe you wouldn't get new data weekly/monthly. Likely the subscription aligns with the data refreshes for most products.
2) It's a one-time fee with a hefty cost attached (I saw some healthcare data sets that were $100K+). You are paying for that in its entirely and just have data rights to it.
Think of the price for data on the prices of stocks. If you want to know what AAPL traded at one month ago, that's free, if you want to know what it traded at 15min ago, that's $25k, and if you want a feed which shows the actual bids and offers, that will run you $2-15mm + infrastructure.
There is tremendous value in cleaning and repackaging this data in easy formats (offering an API would be awesome too). Even better if the provider can offer human support.
Obviously it would be ideal if the government releases all this data in easy formats in the first place, which a lot of governments do, but not all. The second best thing is for some private companies to help, even if they charge for it.
Not your parent commenter, but speaking from experience.
Here is a trivial example : I was looking for a list of all government websites (and their social media accounts), starting with federal all the way to small towns. There are lists, but none complete - at least I couldn't find, and definitely not with their social media handles.
Some people would rather pay and have it all in one place. Some people would pay for you to check its sanity and for having someone to blame if it's wrong.
Sometime it's the corporate paying, and the dev team does not care.
“We live in a data-intensive, data-driven world!”
I know these blog posts are turned out fast, but especially for such a sensitive issue as a world awash in data that no one understands and no one - yet - controls...it seemed like it was “whistling past the graveyard”.