Yes. This was because HiQ was creating fake accounts to scrape member profiles, which was deemed to be in violation of the TOS.
Legal has reviewed our approach in comparison, and we're good.
466 karma · joined July 15, 2016
Legal has reviewed our approach in comparison, and we're good.
The back story is that we started as started as a B2B AMB product. We relied on another provider for data, then got hit with a bill one month that nearly bankrupted us. We ended up building our own system and pivoted to making this data more cost effective for others to access at scale.
Basically, there's still bits and pieces of old marketing material out there that needs to be removed and/or cleaned up.
- https://techcrunch.com/2022/04/18/web-scraping-legal-court/ - https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn - https://www2.staffingindustry.com/Editorial/IT-Staffing-Repo...
Basically, we're addressing the community's feedback and providing general updates on the project. Open to any and all thoughts.
At a certain stage of a company, sales reps expect Salesforce. So much so, that even when we finally caved and got it, I had reps specifically turn off Lightning and stick with Classic mode. It's like Bloomberg - it's what they know and can move fast in.
As much as it pains me to say this, it may make sense to have an option that can mirror the Salesforce UI vs reinventing the wheel. Or maybe even an integration / escape hatch to migrate everything off of Salesforce to this. Basically, if you want reps to adopt it, make it close as possible to what they know.
I settled on doing open source company data: https://news.ycombinator.com/item?id=35977057. I think an open source CRM with the all the world's companies could be pretty killer.
I created an account and will play around with it. Pretty cool so far.
Try disabling your VPN if you're on one, or use a different IP.
To be frank, we went back and forth on this, but in the end, thought the original title was ok. The only other large, "open source" dataset we could find was 9M. So after researching, we came to the conclusion that it sounded clickbaity, but was likely accurate.
And yes on "open source". We fully intend for this to be "open source" in the full meaning of the word, but it seems we were moving too fast and missed adding the formal license.
No, the source code is not available. This dataset is a subset of the raw data our system collects. Our final product made available via the API does a variety of processing steps on the raw data (dedupes, joins, ML predictions, etc). The final, processed data is the piece that is proprietary / subject to the terms.
We will update the site terms to reference this dataset, as we aim to continue releasing an updated version each quarter. I'll have to double check with the lawyers, but it will most likely be MIT licensed.
We're probably going to have to make an public announcement about this...
1. Yes, this is scraped from public sources. 2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here. 3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but the website is not.
We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key company fields like name, industry, size, location, LinkedIn handle, etc. We aim to update it quarterly to ensure that you always have the most up-to-date information.
Disclaimer: Okay, we have to admit, we didn't exactly comb through every dataset out there to verify that ours is the world's largest, but we did our research, and we're pretty sure it might be. Whether or not that's true, we believe this dataset is a robust and invaluable resource for anyone interested in company data.
Related, we built a developer oriented Zapier clone for event scale automations awhile back for our internal product. We've since pivoted and have been debating on potentially open sourcing the engine as well.
We built ours using Rust with a DSL for all the triggers, actions, and action inputs/outputs. The actions themselves are defined as APIs, which makes it easy to add functionality in any language. Most of our actions have been built in Typescript.
Is there interest from anyone in potentially using it?
I replied, "this will eventually result in nearly limitless energy that will fundamentally change humanity".
She responded, clearly disappointed, "oh, okay".
I think for the average person, unless it has an immediate impact on their life, it's difficult for them to visualize future potential benefits.
I was trying to build a B2B sales/marketing app and all the ML stuff we wanted to do required business data, which was too expensive for us. Long story, but after a cofounder blow up, a failed acquisition, then covid being the final straw, I started poking around during lockdown on how to get business data at scale.
I ended up building an API to look up any business by domain/name, which has been particularly popular for fintechs. It's more or less fully automated now and is a nice income stream.
Another path is having a unique story that you can raise money on. A significantly sized problem that you are passionate about and have unique domain experience with. You can potentially acquire this unique experience via strategically picking where to work prior to starting the company. Such as taking a smaller salary specifically to get said experience.
The other path is just busting your ass working for someone else for awhile and building your war chest before going out on your own. Save up at least 1 year of living expenses.
For reference, I've done a blend of #1 and #3. Though in hindsight I wish I did more of #2. Having unique domain expertise gives you a competitive edge.
The impact of that hit me when I was sitting in a taxi at a stop light and watching brick layers work. No mask, no sun protection, and in the middle of rush hour traffic.
I now start with SaaS, and if something becomes strategically important or the costs skyrocket, only then consider an alternative such as building internally or a partnership.
What sort of scale can you guys handle? One of our DBs is in the billions of rows.
I assume we'd need to potentially create custom sources for each destination as well? Or does your system automatically figure out the best "common" schema across all destinations? For example, an IP subnet column.
One note that may be helpful, if all you care about is the HTML, it's better to take a "snapshot" of the page by streaming the response directly to blob storage like S3. That way if something fails and you need to retry, you can reference the saved raw data from storage vs making another request and potentially getting blocked. Node pipelines makes it really easy to chain this stuff together with other logic.
For reference, I run a company that does large scale scraping / data aggregation.
He shared a story with me of a particular lawyer that ends up being paid millions of dollars per hour, because he's able to leverage his connections to help corporate clients jump to the front of the line.
We have it setup with Github actions to automatically deploy to ECS as well.