This article brazenly glosses over HTTPS, which is designed to tackle the very issues that are raised (privacy and authenticity). APIs do not need to sign every response, they just need to properly use HTTPS.
If I call https://api.example.com/?id=123 myself, then I can be reasonably sure that the data has come from the server unaltered.
But what if someone emails me a JSON file? Or someone re-hosts an XML dump? Or a hacker claims to have uncovered proof of...?
As data-journalism gathers pace, it's vital to be able to see whether data is authentic.
But for real.
I had an idea a while back: with all the photo/audio/video manipulation and ‘deep fakes’, there should be a certification available that accompanies (very expensive) cameras/recorders with protected firmware that signs each and every raw frame/recording, so the original is always available upon request from any type of inquiry or investigation.
This seems like a good idea at first but what about whistleblowers or informants?
You've misunderstood the point. The goal is not to identify users. The goal is to sign messages to ensure they were actually outputted by a service.
It's much less painful to abandon the idea that photographs, videos and sound can be ultimately trusted.
https://blog.elcomsoft.com/2011/04/nikon-image-authenticatio...
Same thing happened with Canon's implementation the year before.
I figured it had already been tried.
I wonder if it can still be done.
The comment is fake. The points in it are valid, though.
The big difference here is whether a comment or API response is claiming any kind of authority over certain data, or is a self-contained argument where the validity doesn't depend on where it comes from.
https://developers.google.com/web/updates/2018/11/signed-exc...
It's part of a family of technologies developed by Google to rehost AMP pages without losing the origin.
I know I might use it as a web designer to send interactive snapshots of how the website currently looks/works to the client; instead of using an ephemeral URL and a tunnelling/proxy service.
If a system retrieves data, it should do so with HTTPS, and then it's authenticity can be guaranteed. If it does not use HTTPS (or some similar protocol), then it's authenticity cannot be guaranteed and no amount of signing or hashing can fix that.
If you are not using HTTPS, you cannot just build your own "higher level" version of secure and authenticated communication. HTTPS works because trusted certificates are pre-installed on each of the machines before they start communicating. Without those pre-existing trusted certificates, no amount of encryption, signing, or hashing can guarantee secure and authenticated communication.
At it's core, HTTPS works because there is a distributed system of trust. The principle is that you don't need to directly trust the server that you're communicating with, but you trust another computer that has vouched for the one you are communicating with.
If you roll your own security in any way, at some point you need to send an encryption or decryption key. Even "unhackable" encryption algorithms are worthless if your adversary knows the key. Thus, you need a way to communicate that initial key information securely. Currently, the only way to do that is by (1) manually entering the key on each server, (2) using near sci-fi technology like quantum communication, or (3) by having the keys pre-installed on every machine by the hardware manufacturer or OS provider.
The option that has worked, HTTPS, uses option 3. Maybe you can roll your own encrypted communication system, but you'll either have to rely on manually entering the keys (which in turn rely on other insecure methods like email and phone), or trust the computer's manufacturer to pre-install the keys on the system your using.
There really is no way to "roll your own" system securely without these steps.
And they are already assuming that system 1 and 2 talk using https and system 2 and 3 talk using https.
So they are effectively saying that they do not trust system 2. If that's the case, a signature from system 2 would not help because the signature itself could not be trusted. They would need the signature from the original source, system 3.
That's why they are talking about signature from system 3 along with the payload.
So even if they trust system 2, they can be sure system 2 has not tampered with the message generated by system 3.
No that wouldn't work by itself. System 2 could tamper the data, create their own signature, and send it to system 1.
The only way it could work is if System 1 got the signature directly from System 3, or at least got System 3's public signing certificate. Either way, System 1 needs to communicate directly with System 3 to ensure the data it's getting from System 2 is authentic.
At some point system 1 will need to talk to system 3 to validate the message (unless system 1 and 3 have an already shared secret or public key)
Yes, this is why any post claiming to "roll their own security" is a farce. The current way security works (using HTTPS) is to pre-install trusted certificates on the machine at the hardware manufacturer or OS installation level. The keys that are pre-installed are this initial high level (usually root certificates), can then secure lower level machines, which can further secure even lower level machines. This only works because we trust our hardware manufacturers (e.g., Dell, HP, etc.) to install the correct certificates. These manufacturers are analogous to "System 3" in the example above.
Our cast of characters: Service A, Person 1, and Person 2.
Person 1 makes an API request to Service A. It's done over HTTPS, and Person 1 trusts that the response received is genuine, because it came over HTTPS, and certs were verified and so forth.
Person 1 sends the text of the API response to Person 2. Let's say Person 2 is a journalist, and Person 1 is a potential source. Person 2 doesn't know Person 1, and so logically doesn't blindly trust them. But Person 2 wants to report on some factual thing about the API response. How can Person 2 be sure that Person 1 hasn't modified the API response before Person 2 receives it?
If the API response is signed somehow, Person 2 could go retrieve Service A's public key (directly from Service A) and independently validate that the API response was actually received from Service A, and that it hasn't been tampered with.
I'm not sure I agree with the original author that this is a critical problem that needs to be solved, but that's the gist of it.
The problem here is that Person 1 cannot be trusted, so they can't be the one providing the signature. Even if Person 1 signs, hashes, or encrypts the message when sending to Person 2, all it can prove is that it came from Person 1, not that it came from Service A.
If Person 2 wants to be sure that the message it got from Person 1 actually came from Service A, it will need the signature provided by Service A, not Person 1. To get the valid signature, it will need to contact Service A.
This is a long-winded way to say that if you want to trust what you got, you've got to go to the original source. HTTPS provides tools to do exactly this, and it's special because every time you buy a computer or server, trusted certificates are pre-installed on the machine. Without that initial level of trust, you can't trust anything you receive from anywhere.
That doesn't make sense. The problem described in the article was that the request documents aren't signed, thus there is no way to verify if a response document was actually outputted by a web service. HTTPS handles transport, but has no responsibility regarding request documents. If the problem we need to solve consists of ensuring if a given JSON was actually outputted as a response to a HTTP request to a service (say, if a Twitter message was actually sent by user X on day Y) then how would HTTPS help?
> That doesn't make sense. The problem described in the article was that the request documents aren't signed, thus there is no way to verify if a response document was actually outputted by a web service.
Suppose that you ("System 1") is getting data from an intermediary ("System 2") which in turn is getting data from Twitter itself ("System 3").
Even if the intermediary System 2 signs the data, all that proves is that you got it from System 2. It does not prove that it was not tampered with when System 2 got that data from twitter directly, System 3. It would be easy for System 2 to tamper with the data, and come up with it's own signature.
If you, as System 1, want to confirm that the message you got from the intermediary (System 2) is authentically from twitter itself (System 3), you would need to get the signature from twitter directly. You would need to ask for twitter's signature (System 3's signature), not the intermediary's.
So system 1 will need to contact System 3 in order to validate the information it got from System 2. There's no way around this.
HTTPS is not sufficient for such third-party exchange, but they are necessary.
If you (say "System 1") ask for a twitter feed from an intermediary (say "System 2"), which in turn scrapes the data from twitter itself (say "System 3"), then HTTPS can indeed help. You can't rely on any hash or signature produced by System 2, because they could modify the data and make up a new hash. Instead, you would need to rely on the hash from System 3, to confirm that what you received from System 2 is authentic. Only tools like HTTPS, with their pre-installed security certificates, can do that.
In this case the author criticizes manipulation of the content by third parties. I hope this is not another argument for a digital notary and believe that there are better technical solutions to that problem. Digital ledgers seem to be a niche some people try to establish.
In the end HTTPS is just transport encryption, not content encryption or content signing.
From an architecture point of view I think it is even advantageous if not every services handles HTTPS itself.
In other cases, HTTPS's end-to-end encryption may not be completely desirable - for example, apt does not use HTTPS as it would interfere with simple caching proxies, and it offers no extra assurances than a verified signature does.
What? This is the entire purpose of HTTPS! To do this securely:
1 - you create your server;
2 - you tell a certificate authority (like Symantec, Digicert, or Let's Encrypt) how to find your server, and they do the diligence to make sure you're not trying to redirect another person's server;
3 - they provide you with a private encryption key and expose the public decryption key;
4 - you save the encryption key on your server.
Now, anyone who receives any information from your server can be sure that it came from you because (a) your server is the only one with the encryption key and (b) we trust Symantec, Digitcert, etc to distribute the public decryption key.