The Reconstruction of Torn Documents | Hacker News Reader

The Reconstruction of Torn Documents

bundesarchiv.de

84 pointsjacquesm48 comments

Vernor Vinge's Rainbows End includes this as a minor early plot point; libraries being fed into giant shredders that digitize and stitch together the scraps on their way to the incinerator.

this is the only thing I remember from the book, but I remember it vividly.

> Could shredded paper be reconstructed?

> No. A few thousand other bags contained paper that had been shredded into incredibly small pieces in 1990. This material was destroyed back in 1991 after inspection by the Stasi Records Office. Hence, there is no longer any paper shredded by the Stasi at today's archive .

Shredder chads from the 90s can definitely be reconstructed. This is why the NSA has gradually decreased the maximum size permissible for chads. If they'd held on to that material they could be recovered today.

They absolute can. https://en.wikipedia.org/wiki/DARPA_Shredder_Challenge_2011?...

While not impossible, the volume of the destroyed document would have made this challenging even today. First, you'd have to scan them, then the software would have to be able to handle the massively increased number of combinations from having a whole bag of material rather than three pages.

Given how much of a hard time they're having with hand-torn pages, which should be a long solved problem, it's very likely that it would have been impossible in practice, in the government IT framework for a long time...

Better call Chuck. https://www.reddit.com/r/betterCallSaul/comments/12mniwq/chu...

The answer to the question is still no. The material no longer exists

The full 72 volumes of documents reconstructed from shredded US embassy documents in Tehran.

https://archive.org/details/DocumentsFromTheUSEspionageDen/D...

> Could shredded paper be reconstructed?

Apparently! Vol. 32 of above (per a spot check) has many docs that were shredded. My recommended method is to put papers in a bucket of liquid and then smush it. Good luck reconstructing that.

The amount of people reading a text saying "the material no longer exists" and replying "we can totally reconstruct that" is surprisingly large.

I'm just wondering why the stasi didn't build a large pit somewhere in a land fill, put all the records in it and burn the lot. 16000 bags weighing what 10 kg each is something like 160 tonnes??

Burning paper in bulk for destruction in an uncontrolled process doesn't really work, you end up with huge chunks of unburnt paper because it couldn't get enough oxygen, plus the ashes may be recoverable unless mixed. Especially bad if you only get it halfway done because an angry mob shows up and stops you.

For fast emergency destruction of documents, the NSA has developed a procedure https://media.defense.gov/2021/Jul/02/2002755769/-1/-1/0/EME... where paper and oxidizer get layered in a burn barrel. "The ratio which was adopted was 65 pounds of paper for 100 pounds of nitrate" gives you an idea how much extra oxygen is required for proper combustion.

iirc it works better if you burn the paper in an agitator, like a big wire cage that turns and tumbles the mass of paper over and over until it’s all been stirred and aerated to ashes

>why the stasi didn't build a large pit somewhere in a land fill, put all the records in it and burn the lot.

In hindsight, that would obviously have been the most efficient, but it needs trust and cooperation between regional Stasi offices to give up control of their records.

But The Stasi did not have foresight, was not efficient, did not have cooperation internally. (If they had all those attributes things may have gone differently. So it’s good they were still incompetent at the end)

I watched a documentary about this a few years ago and they spoke about this. At least this is what I seem to remember. The wall came down quickly and caught the Stasi out. Not sure if it was surprise or a desperate hope of holding onto their cushy spy jobs. Anyway, when they realised the consequences of all of their activities becoming public they started to panic. That’s the downside of spying on an entire country in such detail - they had enormous archives. Of course they had a process for deatroying documents but I think most of their business was hording :) All Stasi workers basically tried to destroy as much as possible and as quickly as they could. They specifically mentioned burning (I am fairly confident) - the Stasi people worried that the smoke would give their secret locations away and that the same people pulling down the wall would come for them too. That’s also the same reason they couldn’t drive all the documents away.

No references - this is just what I remember from that documentary.

Just scan all the snippets and upload them. Someone will figure it out.

Scanning them is likely part of the problem.

I think "scanning" these days might be referring to cameras.

Possibly machinery which automatically grabs, straightens, and scans/photographs each of the 400 to 600 million fragments.

The snippets contain sensible data of Stasi victims and often false statements about them. There is a specific law governing the handling of Stasi files, regulating who can get access to what.[1]

[1] Gesetz über die Unterlagen des Staatssicherheitsdienstes der ehemaligen Deutschen Demokratischen Republik (Stasi-Unterlagen-Gesetz - StUG), https://www.gesetze-im-internet.de/stug/BJNR022720991.html (in German)

Wow, fascinating! I hope I’m not the only OCR-minded soul on here that immediately started thinking of pipelines for this one could assemble from all our new DL-enabled toys… it’s a shame that “some American kid” probably doesn’t meet their contracting requirements.

One thing that kinda baffles is that it sounds like the snippets (I bet that German word is fun) haven’t been scanned(/photographed…?) yet. No matter who they contract with, wouldn’t that necessarily be step #1? Or is the scanning the part that needs contracting, and the “e-Puzzler” name was just a lil melodramatic?

For one thing, I feel like it’ll be wildly helpful that they’re on a variety of papers, not challenging like they imply. Once that and bag index have narrowed down neighbor candidates, you’ve got rules and glyphs to go off of, much less recognizable form layouts. It feels like the Vesuvius scrolls but actually tractable…

Scanning 16,000 bags of tangled snippets does not sound that trivial to me. Surely that’s a significant problem and cost on its own.

Let’s say it costs €100 to digitize each bag, that’s only €1,600,000 significantly less than what has already been spent.

That’s an odd price. Not knowing what a “bag” means, I think 13-gallon (“kitchen”) or larger lawn clippings bag is possible.

I wouldn’t for €100 sign up to scan even an office wastebasket (6-8 gallon) bag of torn-up paper. Make it €2000 for that quantity, and we’ll start talking.

I worked in the Pentagon from 1989-1995. I handled “burn bags”, among the other classified stuff I did.

I can’t speak for what their bags are like, but ours were basically large, thick, grocery-size paper bags without handles. Thick enough that they most likely wouldn’t tear, even if they were full and mid-handled somewhat. You rolled down the top and stapled them shut.

Like these: https://live.staticflickr.com/4129/4987369106_ab71780b90_z.j...

But we didn’t shred our documents first. We just put in whole sheets and reams of paper, and we left it up to them to shred first and then burn. But then, that was 1989.

“Each bag contains between 2,500 and 3,500 fragments of torn-up pages.”

Agreed, €100 seems low to process a single bag.

16k bags with 3k shreds each is a few million pieces. You could scan 10-20, or more, at a time spreading a them over a glass sheet, putting other on top and making two pictures. Have a few people doing that and you can process a bag in a few hours. 16k bags for a team that can process 10 bags in parallel is like a year of work.

After that I bet the full reconstruction takes like a month tops with AI matching pieces.

It helps that they used bags; because if all snippets were in one bag, then combinatorially, things would be much more difficult.

n pieces residing in the same b_i, i<16501 bags can be viewed as a blocking function in record linkage. That means while comparisons are O(N^2), here N=16500 and not N=i*16500. In practice, filtering by compatible color (effectively a 2nd-level blocking function) prunes the search space further.

The simplifying heuristic (works not always, but most of the time): "assume that all snippets belong to the same bag".

One important question is how big the individual snippets are - ideally, they'd contain fractions of more than one line of text on top of each other. According to @Pebblebox (see below), “Each bag contains between 2,500 and 3,500 fragments of torn-up pages.”, which means the fragments must be large enough on average to be amenable to language model application.

Then I'd create a data structure with the color of a snippet and its lines of text (either full lines, or prefixes and suffixes at the [left/right] fringes):

  type snippet_type = record
                        snippet_color: RGBColor;
                        text_height:   integer;
                        text:          lines[1..5] of record
                                                        left:  string;
                                                        right: string
                                                      end
                      end;

A statistical or neural language model should be able to piece individual pieces together and come up with a list (sorted by likelihood) of locally probable solutions (called "trellis" or "sausage" in ASR parlance), which can be combined using a beam search.

> Or is the scanning the part that needs contracting

I assume both scanning and reconstruction would be part of the same contract.

There's nothing technical to discuss; where's the meat? This 2012 article at least has pictures: https://www.bbc.com/news/magazine-19344978

I speculate the problem is not technical but financial or political; the ML to do this was up to the task at least ten years ago.

Don't neglect the possibility that there are people in power who don't want these records to come to light. Either because of direct implication of people or their families in the stasi atrocities, "national embarrassment", etc

This was a DARPA challenge back in 2012 too. Shredded documents seem more of a challenge than torn.

https://cacm.acm.org/news/darpa-shredder-challenge-solved/

These days we're reading inside carbonized papyrus scrolls using X-ray tomography: https://news.berkeley.edu/2026/09/16/to-read-2000-year-old-b...

Reconstituting torn-up paper is child's play.

Ah yup the Stasi: a big success of The Socialist Unity Party of Germany, working with their friends in the USSR, to crush dissent (by suppressing speech, imprisoning people, etc.).

I'll just note that today in France there's one TV channel that isn't a leftist one and the left is working hard to make it shut down, under the excuse that it's an evil far-right racist channel. One channel that isn't a socialist (or downright communist) channel and even that is too much: just like the stasi, they want to crush dissent and have that channel closed.

Leftists cannot conceive that people from the other side of the aisle have a right to freedom of speech, so they'll do everything they can to remove it.

That's what the stasi was. And the people with the mindset that made the Stasi possible are hard at work in the world today.

  > Could shredded paper be reconstructed?

  > No. A few thousand other bags contained paper that had been shredded into incredibly small pieces in 1990. This material was destroyed back in 1991 after inspection by the Stasi Records Office. Hence, there is no longer any paper shredded by the Stasi at today's archive .

I find the assertion that it was “impossible” to reassemble shredded documents… disappointing - as they seem to be confusing “very difficult with current tools” with “no way of doing it, ever”

I mean, they made sure it was true by burning them all after they decided it could never be done. Phenomenally short-sighted - unless of course there was a reason they wanted those particular documents to never be read.

> unless of course there was a reason they wanted those particular documents to never be read.

Why would officers from West Germany not want East German documents to be read? That's nonsense. Then they would have destroyed everything, not just th shreddered documents.

Why then would they destroy them, and make sure it’s impossible to ever know their contents. The cost of storage of sacks of paper can hardly have been the reason.

Because they assumed they were already destroyed beyond any possible means of recovery. Would you store a bag of paper pulp/ash that came out of the NSA's pulper/incinerator?

After seen the size of the shreds... Yeah, this doesn't look impossible in the least. These are hand torn pieces, each paper if broken in like 4 or 8.

That's the bags they kept, the sentence is about "properly" shredded bags. Which, again, would be possible in 2011 (DARPA shredder challenge) but it's not that unreasonable that 20 years before that it was considered impossible.

They’ve spent six million euro and counting on this and they are still paying people to manually reassemble them.

Talk about rearranging deck chairs on the Titanic.

> Since the project began in 2007, based on contractual obligations, around 6.5 million euros have been paid to Fraunhofer IPK or the Fraunhofer Gesellschaft

> In the fall of 2013, Fraunhofer IPK was able to demonstrate that the software developed by its scientists, the so-called e-Puzzler, works in principle. However, the process proved unsuited for mass production. Overall, the project never achieved the envisaged automated reconstruction rate of at least 80 per cent.

Well, that was tax-payer's money well spent!

haha right? let's never spend any tax-payer money on anything again! did you know the soviets used a pencil? ha!

This but unironically. Taking money from everybody under the threat of violence and then wasting it should be a hangable offence.

What is up with this cynical comment? They tried to solve the problem and it turned out to be not feasible. Just because they failed doesn't mean trying was bad. I suggest learning about the concept of "expected value".

If you knew anything about the European funding landscape, you'd know that a 6 million Euro project is an enormous amount of money for a project with only one institute involved.

Compare that for instance with the Horizon 2020 project "RePAIR – Reconstructing the Past: AI and Robotics Meet Cultural Heritag", which ran for almost 5 years, had a budget of only 3.5 million Euros and that money was divided among six(!) different partners.

Maybe that gives a little bit of a perspective on "expected value".

2020 is a significantly later date, meaning the fast moving field of machine learning was in a much better state by then. Moreover, the project you mentioned might have been far smaller in scope. The scope of the Stasi document reconstruction is anything but small. They talk about "400 to 600 million fragments which represent around 40 to 55 million pages."

Hard to get too worked up about that when the US govt has allocated about 500 billion to Lockheed Martin for a jet fighter. I'd honestly put the jet and the docs at about the same expected value ("what exactly is this doing for our people?").

"We simply rehired all Nazi's and Stasi's but here there is a sophisticated technical problem to keep you busy."

Reply on news.ycombinator.com