Hacker News Reader

Comment by ManBeardPc | Hacker News Reader

Tons of unstructured data that needs to be cleaned up in so many different ways that are hard to predict. Knowledge distributed in the heads of greybeards, handwritten papers und excel, sharepoint and dozens other locations like ancient proprietary software. Which you have to repeatedly ask for and discover more piece by piece. Then clean up, discover issues, make the customer gather more data/correct it, etc.

View on HN

This knowledge locked in people's heads isn't going to magically show up when you throw AI at it, though.

And relying on AI to clean up unstructured data makes me nervous. Good luck spotting the hallucinations.

Oh yes. Verification needs to be done. Some statistics and validation is absolutely required. Also most customers don’t want their private data uploaded anywhere, so ai gets only test data and writes the script to transform it. As long as it is just XML, CSV or similar it’s pretty reliable. Handwritten notes and bad scans however…

A friend of mine runs a business similar to this for systems migration, which is a fairly tough and tedious "short-put" job (you have to know both old and new system fairly well, and if all goes well, you get the same job done on the new system, while if something goes wrong, you're in a world of pain...):

The AI analyses sample data from old and new system, and then builds a prototype deterministic mapping pipeline. The experts can then inspect said pipeline and with their domain expertise catch edge cases/exceptions/uncovered cases, fix them and improve things, and then you use the vetted deterministic pipeline for migration. Yet, the initial AI draft saves enormous time.

That's definitely how I use AI in general: build the tools instead of process the data. I rather like deterministic behavior.

No, but if we go from data to experiment faster we see earlier if something is wrong. Once we get data we quickly can fix the data pipeline and get results. If there are issues we can quickly show what/why and discuss it with them in detail and make changes based on their response faster.

But the problem identified that is talking to people and having them record data.

Yes, but they tell things and maybe forget to mention certain things or contradict other data they have given you. Now we may not notice it initially until we make an experiment and get the results. Something is wrong and we have to find out what. Talk to the customer again. Fix experiment. Repeat. Faster experiments means the latency from new data to results is less and we finish the project faster. Errors become less expensive and cause less delays. People make errors and data quality is often horrible so the shorter iterations are nice.

But the experiments aren't two weeks of boilerplate. Or if they are, something else is really fucked up in the stack.

It's not all boilerplate of course. But iteration cycle is shorter and the latency matters. We are trying find a formulation, so a lot of experimenting, fine-tuning, checking errors (first via simulations, then real-world with measurements). Then we see what needs to be changed, repeat until error is in desired bounds. But the real-world is messy so we see a problem (lets say adding some certain color does not change the final properties of paint the way predicted) and we go back to the customer, some expert remembers temperature/moisture of the air was different on this day and that affects printing results. Now we need to measure this too and add it to the simulation and add it to the data pipeline. But we may have to wait until the real world has implemented that too (create a process, buy hardware, whatever) and then a while later we finally get real data and can start doing our part. Reducing our latency makes things faster, as you may do something else in between and need to load everything in you head again. The LLM gives results in sometimes minutes: frontend, backend and database. We now have the feature branch where you can upload the data like the customer would do, start the calculation, see the predicted formula. From idea to working prototype. We can focus on making assumptions and checking them quickly. Previously they can make the changes in their Python scripts but getting it in a presentable form took much more work. Now we can go to the customer and present a pretty custom made UI like an end-user would see same day they give us new data or report issues.

The stack certainly could need improvement, but much of the overhead comes from the process being pretty serial because each step depends on the previous. If someone we needs to talk to goes on vacation, sick or has a full calendar this delays it much further. Iteration speed is not linear in such cases. We do a lot of custom solutions and have to implement custom parsers or interface with undocumented ancient software or simply no existing software. Everything is custom, because previously they have no processes, no IT infrastructure or simply only data intended for humans (roughly structured, informally written, ever changing naming schema, typos).

> handwritten papers und excel, sharepoint and dozens other locations like ancient proprietary software

This area is one which AI has shown to be pretty good (as long as some error rate is acceptable). Character recognition, voice transcribing, converting documents from one standard to an other. I can see an argument that if people spent most time doing those things, AI will speed it up (with a bit of added errors).

I am reminded however that in translations and AI generated subtitles, there is still a lot to be desired compared to the human touch. Handwritten papers in a different language, using local short hand and assumptions, may result in some odd and error prone AI transcription.

Reply on news.ycombinator.com