mfrye0

Super interesting. I had the same experience trying to connect a CRM with various tools and the data was typically an absolute mess. It's a hard problem.

Would you be open to chat further? I'm not sure if I can help you or not without learning more, but I'd love to better understand what you're seeing at minimum for research purposes. My email is in my profile.

To be frank, that's the challenge we're working on now. To start, my partner and I were mostly tapping into our network for initial customers. Both of us have been in the Bay Area for 10+ years.

Now, we're working on more of an inbound strategy and have a bunch of ideas. We're debating on cold emailing, but as an engineer myself, I hate getting cold emails.

This hits home. We're building business intelligence APIs around entity resolution, and the buyer/user split gets messy when you have engineering, product, and data science teams all involved.

Engineers immediately understand why matching messy company data is a nightmare, but executives just see delayed projects without grasping the technical complexity.

We're seeing more success lately with "your team burned N months on data matching that should've taken weeks" rather than explaining what entity resolution even is. We're talking to one company right now that's spent 10 years building their own entity resolution system and it still doesn't work well.

But even then, it depends on the company and what they're trying to do.

Interesting. Was that a MDM focused product for healthcare or something more general infra wise like Informatica?

I don't have much context in the healthcare space and the challenges that exist there. We've been mainly talking to people in fintech, supply chain, and sales & marketing, which is primarily where I ran into this at past roles.

Nice! Yeah, a typeahead works to a degree. I imagine that's searching their own instance vs calling out to a standardized DB you manage?

Raw materials is definitely a different animal, so auto-generating definitely works. I know a company where that's all they do - they manually pour over supplier specs to get all the model names.

Agent approach looks super cool. I see the supplier search piece happening there.

We've mapped out ~265M+ businesses globally. We're thinking about this as a data infra angle where products can tap into our system to access all the world's businesses. We're getting requests for processing millions of ERP records to clean/standardize, plus semantic supplier search across our full dataset.

I'll shoot you an email to chat more.

Congrats on the launch! Love seeing modern manufacturing systems.

Do you handle supplier master data management? We're seeing procurement teams struggle with duplicate vendors in their ERPs - same supplier gets entered 5 different ways, messes up spend analytics and supplier relationships.

We're building AI agents for business data cleanup (still in stealth, docs coming). Manufacturing/supply chain customers seem to have the messiest supplier data - way worse than other industries.

Curious if this is something you're thinking about for Carbon? (CTO here, happy to chat)

I'll check that one out.

The copy aspect was the main value prop for the app I chose: Voice Type. You can do ctrl-v to start recording, again to stop, and it pastes it in the active text box anywhere on your computer.

I know what you mean. This new generation of speech recognition is different though. It's able to understand fairly technical terms, specialized product names and other stuff that previously would have been garbled text.

Even if it gets slightly garbled, I often will add a note in my context that I'm using speech recognition. Then Claude will handle the potentially garbled or unclear sections perfectly or ask follow-up questions if it's unclear.

Haha yeah, it does feel a bit weird.

I often work on large, complicated projects that span the whole codebase and multiple micro services. So it's often a blend of engineering, architectural, and product priorities. I can end up talking for paragraphs or multiple pages to fully explain the context. Then Claude typically has follow-up questions, things that aren't clear, or issues that I didn't catch.

Honestly, I just get sick of typing out "dissertations" every time. It's easier just to have a conversation, save it to a file, and then use that as context to start a new thread and do the work.

I checked that one out. The one that Reddit recommended was Voice Type. It's completely offline and a one-time charge:

https://apps.apple.com/us/app/voice-type-local-dictation/id6...

The developer is pretty cool too. I found a few bugs here and there and reported them. He responds pretty much immediately.

My optimization hack is that I'm using speech recognition now with Claude Code.

I can just talk to it like a person and explain the full context / history of things. Way faster than typing it all out.

Yes, none of the top LLMs can do entity resolution well yet. I constantly see them conflate entities with similar names - they'll confidently cite 3 sources about what appears to be one company, but the sources are actually about 3 different businesses with similar names.

The fundamental issue is that LLMs don't have a concept of canonical entity identity. They pattern match on text similarity rather than understanding that "Apple Inc" and "Apple Records" are completely different entities. It gets even worse when you realize companies can legally have identical names in the same country - text matching becomes completely unreliable.

Without proper entity grounding, any business logic built on top becomes unreliable.

Very cool. I read through those links - really sophisticated setup. We're experimenting with something similar on the embeddings side.

Having dealt with this challenge at my last 3 companies, it's easy to hack together something that works most of the time. The hard part is dealing with gnarly customer inputs, the long tail of private businesses globally, and getting close to 100% accuracy (important for legal and risk use cases).

We're building what's essentially an AI-powered version of D&B - combining government registrar data globally with real-time web data at scale. Much more accurate on obscure entities and way faster updates than the legacy providers.

I actually shot you an email - would love to chat more about this if you're up for it.

We're working with an enterprise customer on exactly this problem. The hardest part is entity resolution - figuring out who "Acme Inc" actually is from messy transaction data and what they do.

We built an AI agent specifically for this that's backed by 265M legal entities. Last week it tested 160% better than our customer's existing system on their real data.

Still in stealth but happy to share our API docs if anyone's dealing with this: https://docs.savvyiq.ai/api-reference/#tag/entity-resolution

Open to chat about this problem if anyone wants to connect - email is in my HN profile.

(Disclosure: I'm the CTO)

I was thinking the same. Forward thinking sites might add this, but the vast majority of website owners probably wouldn't be able to figure this out.

Some middle ground where an agent reverse engineers the api as a starting point would be cool, then is promoted to use the "official" mcp api if a site publishes it.

Checking it out and I see you're using CDP.

It's been a bit, but I'm pretty sure use of CDP can be detected. Has anything changed on that front, or are you aware and you're just bypassing with automated captcha handling?

Thanks for the info. That makes sense.

Looking forward to trying out the product more when I have a moment.

Right, I saw that. ChatGPT does the same.

My question is how you can confirm the entity you're referencing in each source is actually the entity you're looking for?

An example I ran into recently is Vast (https://www.vastspace.com/). There are a number of other notable startups named Vast (https://vast.ai/, https://www.vastdata.com/).

I understand Clay, which your Websets product is clearly inspired by, does a fair amount of matching based on domain name or LinkedIn url.

If Websets is doing fuzzy or naive matching, that's okay. I'm just trying to understand the limitations and potential uses cases of your current system.

Congrats on the launch!

How do you dedupe entities, like companies and people? I've noticed ChatGPT tends to provide "great" results when asking about different entities, but in reality it just groups similar sounding entities together in its answer.

For example, I asked ChatGPT about a well known startup. It gave me a confident answer about how much they raised, their current status, etc. When looking at the 3 sources they cited though, it was actually 3 different companies that all had similar sounding names that it just grouped together to form its answer.

Basically, how do I trust the output of your system?

I see the example for running a distributed task queue. The docs aren't so clear though for running a distributed workflow, apart from the comment about using a vm id and the admin API.

We use spot instances for most things to keep costs down and job queues to link steps. Can you provide an example of a distributed workflow setup?

As others have mentioned, converting html to markdown works pretty well.

With that said, we've noticed that for some sites that have nested lists or tables, we get better results by reducing those elements to a simplified html instead of markdown. Essentially providing context when the structures start and stop.

It's also been helpful for chunking docs, to ensure that lists / tables aren't broken apart in different chunks.

Congrats on the launch. This looks awesome.

I'm actually working with a number of companies who are exploring this space. Many of them are in the current YC batch. We're helping to provide the core business data, then we're exploring how we can leverage our scraping infrastructure in other ways to bring costs down.

I'm open to chat if you're interested: michael (a t) bigpicture.io

Perfect. Yeah, we're using both, but mainly TS. We'll test that out.

I've been looking for this exact thing for awhile now. I'm just starting to dig into the docs and examples, and I have a question on workflows.

I have an existing pipeline that runs tasks across two K8 clusters and share a DB. Is it possible to define steps in a workflow where the step run logic is setup to run elsewhere? Essentially not having an inline run function defined, and another worker process listening for that step name.

My thought here is that Google is still haunted by their previous AI that was classifying black people as gorillas. So they overcompensated this time.

https://www.wsj.com/articles/BL-DGB-42522

I think it depends on what you're trying to do. For any given project, what's the goal? Who are you building it for? Are you trying to build a business around it?

There's a big difference in working on a project for fun, and building something to solve a problem for users. In the article, I do not see any mention about talking to users, customers, etc. and asking what they want. It's a lot more motivating to continue working on a project when you're building something people want and solves real problems.

Pretty small still at 500gb. It only stores hot data right now and a subset of what's important. Most of our data is in S3.

If you're already on AWS, I recommend switching to postgres for now. For context, I have 3 RDS instances, each multi zone, with the biggest instance storing several billion records. My total bill for all 3 last month was $661.

Postgres has full text search, vector search, and jsonb. With jsonb you can store and index json documents like you would in Mongo.

- https://www.postgresql.org/docs/current/textsearch.html - https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-rd...

I know a guy that built his whole product out on Hubspot using their free CRM and APIs. I believe he said he's storing a few hundred million records there and there's been no issues so far.

I used to work for a division of State Farm insurance which ran into some similar issues with addresses on or close to state lines. Insurance is also one of those businesses that's very particular about location, as we were only approved to do business in certain states.

The solution we ended up going with was to use the free Tiger shape files from the US Census (https://www.census.gov/geographies/mapping-files/time-series...), which enabled us to statically "lock" our geocordinates. Apart from filing our approach and getting it approved, the logic was that we could always point back to it being a government database if there was ever a problem.