HNHacker News
TopNewBestAskShowJobs

yfontana

172 karma · joined June 15, 2024

submissionscomments
yfontana··on PDF to Text, a challenging problem
I've been working on extracting text from some 20 million PDFs, with just about every type of layout you can imagine. We're using a similar approach (segmentation / OCR), but with PyMuPDF.

The full extract is projected to run for several days on a GPU cluster, at a cost of like 20-30k (can't remember the exact number but it's in that ballpark). When you can afford this kind of compute, text extraction from PDFs isn't quite a fully solved problem, but we're most of the way there.

What the article in the OP tries to do is, as far as I understand, somewhat different. It's trying to use much simpler heuristics to get acceptable results cheaper and faster, and this is definitely an open issue.

yfontana··on Universe expected to decay in 10⁷⁸ years, much sooner than previously thought
We don't know for sure that the universe is a closed system.
yfontana··on Evidence of controversial Planet 9 uncovered in sky surveys taken 23 years apart
Primordial black holes are black holes that formed right after the big bang. Basically areas where gravity caused the extremely dense matter of the universe's first instants to collapse into black holes before expansion could pull it apart. Their existence has been hypothesized but not confirmed (or definitively rejected) so far.
yfontana··on France's new high-speed train has Americans asking: Why can't we have that?
It's absolutely not crazy, and I don't know how a car could only cost you 1k/year. This is just an example but has the average cost of motoring in Ireland in 2019 at almost 11k€/year : https://www.theaa.ie/motoring-advice/cost-of-motoring/.

One of the issues with trains is that people often severely underestimate the true total cost of car trips.

yfontana··on Hacking the call records of millions of Americans
A while ago I worked on a system handling call records for a large telco. Call records were considered sensitive information at that company, and distributed only where definitely needed. I'm sure security wasn't bulletproof, but there were regular audits to check that employees and contractors didn't store records in places they weren't supposed to.

One of the main functions of the system that I worked on was to create various anonymous and/or aggregated versions of the data, which could be distributed and used more widely (for stuff like fraud detection, network provisioning, marketing...).

yfontana··on xAI has acquired X, xAI now valued at $80B
This is inaccurate. Reserve requirements are not the only cost that banks have to pay for the money they create. If bank A lends $20B, and the borrower spends it to buy something from someone with an account at bank B, bank B isn't going to accept $20B of "made-up" bank A money, it'll want $20B of central bank money from bank A. If bank A doesn't have that that $20B of central bank money, it'll have to borrow it from the central bank, at the cost of the reserve interest rate. Being able to transfer central bank money to bank B is essentially what backs the money that bank A created.

(in reality some of that $20B would probably make it back to bank A as money circulates, so the actual amount of reserve money that banks have to borrow depends on more complicated factors. This also doesn't talk about what happens if the borrower defaults. But the fundamental principle remains the same, and is what ultimately limits how much money a bank can lend).

yfontana··on War story: the hardest bug I ever debugged
I think the hardest bug I've had to work on was just plain irreproducible. Once we exhausted all other ideas, we just attributed it to some sort of bit flip. Not a very satisfying resolution, but kinda cool to have encountered such an issue at least once.
yfontana··on Mistral OCR
I'm working on a projet that uses PaddleOCR to get bounding boxes. It's far from perfect, but it's open source and good enough for our requirements. And it can mostly handle a 150 MB single-page PDF (don't ask) without completely keeling over.
yfontana··on MongoDB acquires Voyage AI
This may be a shock to many HN readers, but MongoDB's revenue has been growing quite fast in the last few years (from 400M in 2020 to 1.7B in 2024). They've been pushing Atlas pretty hard in the Enterprise world. Have no experience with it myself, but I've heard some decently positive things about it (ease of set up and maintenance, reliability).
yfontana··on The possibilities for dark matter have shrunk
>>> PBS Spacetime has a video out now on whether gravity is actually random. I don't know whether it addresses theories of superfluid quantum gravity.

It doesn't, it talks about the possibility of randomness allowing gravity to be classical and not quantum. If my understanding is correct, the gist of it is that certain random fluctuations in gravity would prevent it from contradicting Heisenberg's uncertainty principle when interacting with quantum entities.

yfontana··on The possibilities for dark matter have shrunk
As written in that article, there are questions to be answered before this observation can be considered a serious challenge to dark matter. Such as: why don't we observe the same phenomenon in other galaxies? Why are things orbiting the Milky Way farther out (such as the Magellanic clouds) not affected either?
yfontana··on Lidl's Cloud Gambit: Europe's Shift to Sovereign Computing
It's not even all tech either. Europe has some big players in silicon or biotech for example. It does however lack giants that introduced major disruptions in the way things are done, like cloud services, social networks, gig economy etc.
yfontana··on Shapez2 Released
To be clear it's an early access release, although as far as I can tell it's very high quality early access, with a lot of polish and heaps of features.
yfontana··on Classifying all of the pdfs on the internet
I've been doing some work for an infrastructure company as well. They have a total of about 1 billion pages of PDF documents in their archives. If we assume even just 30 KB per page (which is quite low, all the PDFs I just randomly checked were higher, sometimes quite a bit so), that's already 30 TB of PDFs, just for that one company with 1B in annual sales.
yfontana··on Markov chains are funnier than LLMs
LLMs were never very good at directly generating original jokes, for a simple reason: writing a good joke generally starts with finding a good punchline, and then setting it up. An LLM generating token after token will first write a set-up, and then try to shoehorn a punchline into it. Prompt engineering can fairly easily work around this, but just straight-up asking an LLM for a joke never really produced good results on average.
yfontana··on Building data infrastructure that will last
Maybe better recognition of data architecture as a specific domain would help. I'm a data architect, I have a specific set of skills to build and maintain data platforms, but when applying for jobs I'm very often finding myself in 1 of 2 situations.

1) I'm interviewed by software developers / architects, who expect me to have strong knowledge of software development patterns (like OOP patterns or microservices design in webapps). Well I know the basics but not at senior level. That's also not what's actually needed to design and implement a data platform.

2) I'm interviewed by data engineers, and I'm expected to show expert knowledge in data pipelines implementation. That's a lot closer to what I can do, but it's still not the core of my skill set, and it's a bit frustrating not to be judged on that.

yfontana··on Building data infrastructure that will last
In many cases, that means gaining the freedom of creating a huge mess (speaking from experience)
yfontana··on Show HN: Zerox – Document OCR with GPT-mini
My testing was admitedly limited (but focused on cases where GPT-4o was struggling), but in my experience Claude 3.5 also produced fewer hallucinations than GPT when doing OCR with figures / tables recognition. It is of course quite pricey as well.

However both GPT and Claude still struggle with documents that are very heavy on infographics, figures and tables.

← PreviousPage 3 of 3