Hacker News Reader

Comment by AznHisoka | Hacker News Reader

What makes a dataset hard/expensive to obtain?

I had a friend tell me recently about a client using commercial real estate data for lead gen. He mentioned https://compstak.com/

Basically, identifying companies that are doing well / expanding by how big the space is they leased. This sort of data is apparently very hard to get, but gives users a competitive advantage.

Real estate data, and companies like compstak are exactly the kind of niche markets I'm talking about. Agents are willing to spend large sums to get access to this data and it can be resold multiple times. Unfortunately it is also a market full of existing competition with some established players.

What other markets for data are similar? In general, data that leads to prospect generation is desirable because sales agents are willing to spend money to make money. Are there any other markets like that?

So it sounds like the salient aspect here isn't necessarily the type of data, but the manner in which that data is collected. Looks like compstak's success is a result of creating a platform that facilitates crowdsourced data points that are difficult to acquire using traditional data collection approaches...that scarcity is what makes the data valuable, especially since that data can be used for leverage in a negotiation. Also, they appear to prop up the overall scarcity by only granting access of existing data to users who provide new data.[1]

I'm curious how they figure out how much to charge companies for this data? And also how they stop real estate insiders from gaining access without sharing new data?

[1] https://techcrunch.com/2012/10/18/compstak/

The medical markets, but I don't want to go any further because that's what I'm doing right now :p

SMART on FHIR is a newish standard for medical applications that is getting a HUGE push from large companies like Cerner, Epic, along with all the tech giants. Hospitals are itching for more FHIR apps that can integrate directly into their Electronic Health Record system (and web apps be delivered directly on a doctor's web portal within the hospital's IT system).

So that might be a good place to start poking around...

Here's a good brief overview: https://healthtechmagazine.net/article/2018/10/everything-yo...

A really good example of this is labeled medical imaging data.

Some key contributing factors: multiple stakeholders & consent/approval issues, legal & technical constraints on access, depending on the application labeling may only be possible using very expensive experts. Lot's of human interaction.

I agree it is expensive to gather but is it something that can be sold at high cost to 1000s of customers? It seems the market for purchasers of that data might be limited to a small number of companies, probably hoping to build ML models.

The question I responded to was what made it hard and/or expensive to obtain, which I think I answered.

Commercial viability of doing so for profit is a different issue, but I see that's the other part of your original comment. It's not an obvious answer, partially because there are a lot of different scenarios within that blanket "medical imaging", and what the putative customer might want to do with it.

Yes, I should have followed the thread better. My mind is focused on a particular kind of commercial viability which is niche markets (in the 1000s to 10,000s) willing to pay for access to data.

One aspect I know of is internal private-business analytics. For example:

- How many truckloads of widgets did the widget company ship out of Warehouse A compared to Warehouse B in 2019 vs 2018.

- What is the purchase ratio of titanium to steel for Company X over the past 5 years?

This type of data is valuable to seek out emerging trends, risk minimization, stock analysis, etc. Very hard to find legitimate data on your own.

One example is that it could require a significant amount of human legwork - e.g. Google street view. Another example is it might require significant dev effort to clean and combine several raw data sets into a refined output data set.

The costs of digitization and annotation.

Reply on news.ycombinator.com