Let me know what I should improve.
9,168 karma · joined June 23, 2019
Also made HN Wrapped (hn-wrapped.kadoa.com)
Happy to chat about data, AI, startups, or anything else: adrian@kadoa.com
Let me know what I should improve.
Kadoa is an AI platform that automates unstructured data ETL.
80% of data is unstructured and traditional methods such as web scraping, PDF parsing, and manual data preparation require expensive engineering resources and constant maintenance.
We’re solving this at scale, but we can't do it alone. We are looking for talented people who share our passion for crafting exceptional software products.
There is one official role, but we welcome conversations with anyone interested in joining the team.
- Frontend Developer/UX Designer: https://www.kadoa.com/careers/frontend-dev-ux-designer
If you want to iterate to greatness and work at the intersection of data and AI, please email us at (adrian at kadoa dot com) and mention HN in the subject line.
Good bots: search engine crawlers that help users find relevant information. These bots have been around since the early days of the internet and generally follow established best practices like robots.txt and rate limits. AI agents like OpenAI's Operator or Anthopic's Computer Use probably also fit into that bucket as they are offering useful automation without negative side effects.
Bad bots: bots that have a negative affect website owners by causing higher costs, spam, or downtime (automated account creation, ad fraud, or DDoS). AI crawlers fit into that bucket as they disregard robots.txt and spoof user agent. They are creating a lot of headaches for developers responsible for maintaining heavily crawled sites. AI companies don't seem to care about any crawling best practices that the industry has developed over the past two decades.
So the actual question is how good bots and humans can coexist on the web while we protect websites against abusive AI crawlers. It currently feels like an arms race without a winner.
The best products will be defined by everything "non-AI", like UX, performance and reliability at scale, and human-in-the loop feedback for domain experts.
I'm using this to test the humor of new models.
The real challenge is building reliable and accurate ETL pipelines (document ingestion from web, OCR, classification, validation, etc.) that work at scale in production.
The best products will be defined by everything "non-AI", like UX, performance, and human-in-the loop feedback loop for non-techies.
Avoiding over-reliance on specific models also helps. With good internal eval data and benchmarks, you can easily switch or fine-tune models.
Isn't game development already discussed here?
I actually built a side project that categorizes front page articles so I can filter for topics. Here's an example for recent gamedev content: https://www.kadoa.com/hacksnack/d57360e8-1eb1-4800-a711-f0d5...
Yes it's a lot of risk, yes most startups fail, and yes you could make more money at FANG. However, this is one of the most exciting times to build a startup, thanks to AI.
The big winners of the all the AI advancements (especially OS) are devs and AI startups:
- No more vendor lock-in
- Instead of just wrapping proprietary API endpoints, developers can now integrate AI deeply into their products in a very cost-effective and performant way
- Price race to the bottom with near-instant LLM responses at very low prices are on the horizon
It's as if your product automatically becomes better, cheaper, and more scalable with every major AI advancement. This leads to a powerful flywheel effect: AI improves product -> better product attracts more users -> users generate more data
Besides the product perspective, AI also allows you to operate your startup a lot more efficiently.
Will we see a single-person unicorn soon? Very likely not.
Will we see very successful small and bootstrapped startups? Yes, already the case.
What's a reasonable way forward to deal with more bots than humans on the internet?
Some challenges I faced:
- Warranties have many nuances depending on the country and the fine print
- Monetizing this is hard. I see your using affiliate links, but this will hurt the trust of your users.
- Only very few users are willing to write valuable in-depth reviews. I tried to collect data on how products perform over their lifetime (https://www.looria.com/reviews), but never reached a critical mass.
- Astroturfing is a major issue. I aggregated many reviews from Reddit, but even there it's getting worse.
Happy to chat or help out with data if you're interested :)
Unfortunately, it looks like the project was abandoned after the initial talk and paper in 2020.
I assume that we'll get there soon with small LLMs running directly in the browser (like Gemini nano) that will make it easy to query/structure website data.
We are building the foundational infrastructure for unstructured data ETL.
Our platform enables any organization to instantly extract, transform, and integrate unstructured data (websites, documents, emails, etc.) - without code or maintenance overhead.
You’ll work at the intersection of AI, data engineering, and software development, shipping code with real-world impact at industry-leading customers.
There are two official roles, but we welcome conversations with anyone interested in joining the team.
- Senior Software Engineer: https://www.kadoa.com/careers/senior-software-engineer
- DevOps Engineer: https://www.kadoa.com/careers/devops-engineer
Please email us at (adrian at kadoa dot com) and mention HN in the subject line.
I'd love to spend my time working on such articles when I'm retired :)
The big winners: we developers.
- https://www.ycombinator.com/companies/tableflow
- https://www.ycombinator.com/companies/reducto
- https://www.ycombinator.com/companies/mindee
- https://www.ycombinator.com/companies/omniai
- https://www.ycombinator.com/companies/trellis
At the same time, accurate document extraction is becoming a commodity with powerful VLMs. Are you planning to focus on a specific industry, or how do you plan to differentiate?
I want to get some broader feedback before completely switching my workflow to Aide or Cursor.
Here is what we ended up with:
- Extraction: We use codegen to generate CSS selectors or XPath extraction code. This is more efficient than using LLMs for every data extraction. Using an LLM for every data extraction, would be expensive and slow, but using LLMs to generate the scraper code and subsequently adapt it to website modifications is highly efficient.
- Cleansing & transformation: We use small fine-tuned LLMs to clean and map data into the desired format.
- Validation: Unstructured data is a pain to validate. Among traditional data validation methods like reverse search, we use LLM-as-a-judge to evaluate the data quality.
We quickly realized that doing this for a few data sources with low complexity is one thing, doing it for thousands of websites in a reliable, scalable, and cost-efficient way is a whole different beast.
Combining traditional ETL engineering methods with small, well-evaluated LLM steps was the way to go for us