Underestimating what an organization like NSA has the resources and imagination to do, is a chronic ailment.
https://archive.fosdem.org/2014/schedule/event/nsa_operation...
2,337 karma · joined June 15, 2010
https://archive.fosdem.org/2014/schedule/event/nsa_operation...
You mean like any human rights organizations who send people to authoritarian states ?
You mean like any Foreign Ministry sending a courier out in the world ?
You could learn so much about operational security, if you wanted to, just by reading open sources.
Can I recommend you start with a wonderful old article called "A first tour like no other" ?
If your adversary model is a wild-eyed gun-slinging mid-west racists and you are black, then encryption is not going to be a factor in your death, and speculating what you can or cannot convince them about is besides the point.
If a sane adversary has captured you and your devices, say border police in some police-state like Belarus, Brazil or USA, it would be silly to assume that you know more about disk-encryption than they do.
Most importantly, you would be very silly to assume that they will not simply lock you up, until you provide access to the device, aka "XKCD538-lite".
There are people who have languished in hell-hole jails for years already, not because they are unwilling to provide access, but because they cannot provide access, but are unable make a convincing showing of that.
Competent organizations make sure their travelers can make that showing convincingly, and one of the steps they take, is to make 100% sure there is no unexplained high-entropy data on their devices.
Imagine ending up in a foreign jail for years, just because you once deleted a huge gzip'ed file, and the first sectors subsequently got overwritten ?
Yeah, that happened: "Now decrypt this other secret partition!"
It's even worse with SSD drives, where the "hidden reserve" many times larger.
Wiping is of course better than doing nothing, but neither lives up to the requirements of GDPR.
Where wiping disks is sufficient, for instance "internal reuse", it is always some number of specified writes, followed by writing all zeros, so both the donor and receiver can verify the disk as empty.
Note that wiping a SSD with all-zeros is likely to be optimized out by its embedded controller.
IT-Dude-Syndrome is when IT-people come up with scenarios which hinge on "IT-Dude" being smarter than everybody else in the whole world.
First, if you're suspected of anything in a law firm, you're out until the issue is resolved, you don't get to "destroy old laptops" or anything of that sort.
In a high-security operation you don't even get to touch a laptop, and you will be handed a loaner-phone, while your own phone is turned off, signed, sealed and stored in a safe until the matter is resolved.
Second, if "standard procedure" in that law firm is to "wipe the drives before selling them", that law firm is not following best, or even minimum, security practices for the business they are in, and that is what the whistle-blower should report to the relevant authority.
Third and most significant, IT-dude is smarter than the obviously junior IT-dude called in to "check the drives" IT-dude "were supposed to wipe" ?
You know what, he's not.
In such scenario you can be damn sure, that whoever is called in to check is several levels smarter than IT-dude.
And you know what super-IT-Dude will do when she see the disks full of high entropy data ?
She will point to the final step in the official procedure for wiping disks, which is to write all zeros to the entire media to prove that there is nothing hidden.
She will write in the report that IT-dude did not follow a trivial procedure to the letter, and point out that the most likely, in fact the only credible, explanation for skipping a trivial step in the procedure, was that IT dude were trying to exfiltrate data.
Being able to prove that you cannot possibly decrypt the partition, moves you out of the XKCD 538 scenario.
That is a "much less negative" outcome.
If the adversary does not belive you, their best bet is to keep torturing you, until they get their way.
What you want is "Proof of inability to decrypt" so that the adversary has nothing to gain by waterboarding you.
The only problem OP has solved is detecting overwriting of metadata inadvisably comingled with the user data.
That problem does simply not exist in phkmalloc, or jemalloc for that matter, writing past the end or before the beginning of an allocation does not automatically give you the ability to violate the internal integrity of the malloc implementation.
If you actually read the first link you pasted carefully, you will see what an amazing tour-de-force it was to exploit CVE-2003-0015, which was a run-of-the-mill double-free mistake in CVS.
As I remember it, there were proof-of-concept exploits against list-based mallocs in a matter of days, it took months before bbp's article appeared.
When I wrote phkmalloc, most of the exploitable malloc-related problems were usage problems, double frees, freeing modified pointers, writing past start or end of allocation etc.
If you had seen my presentation in New Orleans 1998, you would undoubtedly remember my handwritten slide with around 20 setuid programs phkmalloc had found issues in within the first year.
That is the primary reason why there are so few malloc(3) related exploits these days: phkmalloc and later jemalloc complained about them.
Those are mistakes which list-based mallocs do not find, because it is too expensive to walk the list all the time, and on top of that, they run slower, because the list is O(n)!
There is neither a reason, nor an excuse for using a list-based malloc in this day and age.
Neither OP nor I claim to have solved the "big" malloc problem, because, as I already explained: That takes hardware support.
In addition to all the security problems inherent in mingling metadata directly next to user-data, the linked list gives O(Nalloc) performance on free(3) and realloc(3) whereas my malloc had O(1) performance.
That may not seem like a lot, but when the 'O' is a page-in from disk, and Nalloc is from C++ code, it is nothing to sneeze at.
Not only did that make my malloc faster, but it would, not "could", but "would" detect several classes of malloc-usage errors, double-free etc, unconditionally.
Over the next 10-ish years, that practically eliminated entire classes of malloc-mistakes, making a lot of FOSS software safer, no matter which malloc you used it with.
Given the definition of the malloc(3) family API, there is no way you can detect all corruption without hardware support, but there are people working on that too, notably Robert Watsons CHERI project at Cambridge.
So yeah, nice pointer arithmetic, but how about people solved the real problem instead ?
(On big memory and multi-core systems use jemalloc, on small memory and single-core systems use phkmalloc.)
https://papers.freebsd.org/1998/phk-malloc/
(And got better malloc(3) performance at the same time.)
Want to bet if they are going to make all the same mistakes themselves, or if they are willing to learn from history?
There are several articles about that in BSTJ, with pictures.
Here is a 1934 article as an example, long before anybody had even named microwaves:
https://archive.org/details/bstj13-4-532
75Ohm gave the lowest loss and thus the longest distance between repeater-amplifiers on the trans-continental carrier-telephony coax-cables.
These same cables (and their brethern: microwave links) also carried television, thus enabling the first "television networks" as opposed to "television stations".
In many cases even the stand-alone television stations paid AT&T to connect their down-town studios to the hill-top transmitter.
So television uses 75Ohm because AT&T did.
And AT&T did to minimize the number of repeater amplifiers across USA, because 75Ohm has the lowest loss.
Ohh, right: https://queue.acm.org/detail.cfm?id=2508864
Keep your options open by keeping your code portable.
Maybe have them download and sort a 32GB data set on a 64GB SD card on a RPi3 board ?
The smart ones will realize that downloading can be their first pass over the data, and all of them will be befuddled by the interesting performance anomalies when photo-grade SD-cards go on sabaticals, if you annoy their simplistic flash-adaptation layers.
If they characterize "the trouble", they may find that obsolete sequential algorithms for tapes run faster than random access algorithms in that environment.
...which is precisely the kind of thing companies expect a CS-graduate to be good at.
This article seems to pop up about every other year, and pretty much garner the same general reactions every time.
A lot of CS-majors immediately go in to defensive mode, starting to spew O(blablabla) formulas while thumbing their text-books.
That actually just underscores the main point I tried to communicate in that article: CS education simplified the computing platform used for algorithmic analysis so much that it has (almost) lost relevance.
The big-O has little to do with real-world performance any more, because of the layers of caching and mapping going on. In practice O(n²) can easily be faster than O(n) for all relevant n, entirely depending on ordering of access in the implementation.
Also, CS-education tends to gloss over important footnotes. Take quicksorts worst-case performance, which even wikipedia describes as "rare". It is not. Sorted data are very, very common.
Originally researchers were more careful about these issues, and they operated with several distinct big-O's. Re-read TAoCP about sorting on tape-drives for a classic example.
Some pull out "Cache-Oblivious Algorithms", usually having never actually tried to implement, benchmark or use any of them.
They are, as a rule, horribly complex, and seldom faster in practice, because they hide a very large performance constant in their complexity.
More often than not, there are no available reference implementation, so people cannot easily try them out, and none of them are fast enough to compensate of the embuggerance of the software patents which cover them.
There are usually one or two who claim that "Virtual Memory" is no longer a relevant concern and that one should just put "enough RAM" in the machine.
Throwing hardware at the problem is a nice job if you can get it, and a nice hobby if you can afford it.
However, even if the money is there, what about the energy consumption ?
Our climate is rapidly deteriorating because of greenhouse gasses from our energy-production, and more and more algorithms run on battery-power.
If I were writing a CS-thesis or CS-textbook today, it would be all about big-E() notation, and how it only weakly correlates with big-O() notation.
I worked on a SW-project there in the late 1980-ies (Moving the European Parliament from OSI to TCP/IP) and it was a strange and wonderful place indeed.
And yes, very good food, I gained four kg in the first two months :-)
But there were absolutely no doubt anywhere that there were two classes of people: Italians and "Strangero" like me, and God help you if you did not understand Italian...
The sadness of Olivettis passing is somewhat tempered by the fact that Arduino grew from its ashes.
We have never compared the C900 physically to the A2000 in datamuseum.dk, we should do that some day.
Commodore was a very serious and successful supplier of small business computers with the 8xxx and 82xx series.
The main problem was that in this segment Commodore came out of a typewriter/calculator mindset, so there were absolutely nobody who ever thought "lets connect computers together", and that set a hard upper ceiling how how large companies they could sell to.
Once a business grew, the CBM computers were replaced, not because they were unhappy with them, but because they were not big enough.
Heck, even at Commodore we used IBM S/34 internally.
The lack of a vision going forward also caused local projects to suck up too many resources, Germanys PCs, the C16/+4 disaster and so on.
The 900 was very much a US project, Germany wanted the PCs.
The C900 prototypes were manufactured in Germany however, but that was simply because they had the facilities for the large PCB's from the 8xxx series.
It is natural to have very expensive checks in development and test-environments, they help and aid development and QA to no end, and of course they should be disabled in production.
But the run-of-the-mill assert which documents integer ranges, pointer non-null-ness, data structure relationships and so on, cost nothing in performance and should be left in production code.
And as I said: We developed Varnish Cache to prove this point, and we overwhelmingly have: One bad CVE since 2006, and "only" about 20% of all web-traffic goes through Varnish at some point or other.
Re Huawei:
The report claims that there where no checks at all, and I'm saying they have not documented evidence for that claim, because the lack of checking code could be either disabled asserts in production code or the compiler reducing it as dead code.
As for disabling asserts in the code you ship: That is such a quaint 1980'ies thing to do, and people should have stopped doing it long time ago.
First, asserts are usually free, in the sense that the values checked are already in registers.
Second, any assert the compiler can evaluate is free, because no code will be generated.
Third, asserts can often enable the compiler to produce better and faster code, because it knows more about the values it produces code for.
And this is not theory: Nobody has ever accused Varnish Cache of being slow, despite the fact that approx 10% of all source lines are asserts which cannot be compiled out.
My money is on Huaweis function doing the check with an assert which was compiled out for deliverable code.
Why else would the have the function in the first place ?
The fact that code is generated for the function as shown, indicates that it was not a static inline function, because the compiler would have reduced that to just the regular memcpy call, so it was probably in a library.
So yeah: Industry Best Practice Incompetence.
Take the "VOS_memcpy_s" function they condemn for not checking one of the arguments on page 35.
If the source contains a check that the destination length is no less than the source length, for instance in an assert(), and the compiler is able to conclude that will always be the case, it is perfectly justified in not generating code for the check.
The trouble is that the check code would also be missing if the "production code" was compiled with asserts disabled.
To make the conclusion they do, they have to have checked that:
A) All implementations lack the check and all are in not-link-time-optimized library locations. (in which case asserts was probably disabled)
or
B) All distributed implementations (because VOS_memcpy_s was either static inline or #defined) all lack the check and at least one of them is called with arguments where the check does not hold or cannot be determined at compile time. (in which case asserts was probably also disabled.)
If they cannot prove either of these two points, it may just be that the code is in fact correct, and that the compiler was able to determine this and did not need to emit the checks.
For all their bragging about how state of the art their tools are, I see no indication that they have analyzed that deep and comprehensive.
That said, Huawei's code is still has atrocious quality.
Or as the C-team calls it: "Industry Best Practice"
It is similar how you can move constitutional questions from state to federal court in the USA, with the main difference being that EU does not have federal circuits of courts like USA does, but effectively only a supreme court.
You have reinvented the Mainframe Channel Processor!
Your next challenge: Try to avoid reinventing the 3745 Frontend Processor.