API responses should be signed | Hacker News Reader

API responses should be signed

shkspr.mobi

40 pointsedent69 comments

This article brazenly glosses over HTTPS, which is designed to tackle the very issues that are raised (privacy and authenticity). APIs do not need to sign every response, they just need to properly use HTTPS.

The author seems to be aware that https has this function as they seem to state using it ensures the response they see is valid. However, they don't seem to understand why that is or that the raw response would remain verifiable to others.

This is addressed in the article specifically. They're not talking about the person making the API request needing a signature to verify its authenticity. It's about if someone else sends you an API response that they retrieved, you should be able to verify it wasn't tampered with.

> They're not talking about the person making the API request needing a signature to verify its authenticity. It's about if someone else sends you an API response that they retrieved, you should be able to verify it wasn't tampered with.

If a system retrieves data, it should do so with HTTPS, and then it's authenticity can be guaranteed. If it does not use HTTPS (or some similar protocol), then it's authenticity cannot be guaranteed and no amount of signing or hashing can fix that.

Most access to HTTPS is transparent, i.e. signatures are not that easy to extract and independently verify (not to mention all the issues with CA infrastructure)... Basically, while maybe you could technically accomplish it with HTTPS, there are huge usability gains to be had in exchange for the inefficiency of duplicating some lower-level functionality at a higher level.

> while maybe you could technically accomplish it with HTTPS, there are huge usability gains to be had in exchange for the inefficiency of duplicating some lower-level functionality

If you are not using HTTPS, you cannot just build your own "higher level" version of secure and authenticated communication. HTTPS works because trusted certificates are pre-installed on each of the machines before they start communicating. Without those pre-existing trusted certificates, no amount of encryption, signing, or hashing can guarantee secure and authenticated communication.

I agree that from a practical perspective, the best way to secure one’s communications with a web server is to enable it with HTTPS. I also think though that it’s possible, for any number of reasons, that one might want signed API responses independently of that (e.g. OTA software updates). Once you get into cryptographic identity though, you’re back to CA/PKI all over again...

> it's s possible, for any number of reasons, that one might want signed API responses independently

At it's core, HTTPS works because there is a distributed system of trust. The principle is that you don't need to directly trust the server that you're communicating with, but you trust another computer that has vouched for the one you are communicating with.

If you roll your own security in any way, at some point you need to send an encryption or decryption key. Even "unhackable" encryption algorithms are worthless if your adversary knows the key. Thus, you need a way to communicate that initial key information securely. Currently, the only way to do that is by (1) manually entering the key on each server, (2) using near sci-fi technology like quantum communication, or (3) by having the keys pre-installed on every machine by the hardware manufacturer or OS provider.

The option that has worked, HTTPS, uses option 3. Maybe you can roll your own encrypted communication system, but you'll either have to rely on manually entering the keys (which in turn rely on other insecure methods like email and phone), or trust the computer's manufacturer to pre-install the keys on the system your using.

There really is no way to "roll your own" system securely without these steps.

OP is talking about system 1 getting a response generated by system 3 VIA system 2 (potentially trusted man in the middle).

And they are already assuming that system 1 and 2 talk using https and system 2 and 3 talk using https.

> OP is talking about system 1 getting a response generated by system 3 VIA system 2 (potentially trusted man in the middle).

So they are effectively saying that they do not trust system 2. If that's the case, a signature from system 2 would not help because the signature itself could not be trusted. They would need the signature from the original source, system 3.

> a signature from system 2

That's why they are talking about signature from system 3 along with the payload.

So even if they trust system 2, they can be sure system 2 has not tampered with the message generated by system 3.

> That's why they are talking about signature from system 3 along with the payload. So even if they trust system 2, they can be sure system 2 has not tampered with the message generated by system 3.

No that wouldn't work by itself. System 2 could tamper the data, create their own signature, and send it to system 1.

The only way it could work is if System 1 got the signature directly from System 3, or at least got System 3's public signing certificate. Either way, System 1 needs to communicate directly with System 3 to ensure the data it's getting from System 2 is authentic.

I think you are right.

At some point system 1 will need to talk to system 3 to validate the message (unless system 1 and 3 have an already shared secret or public key)

> I think you are right. ... At some point system 1 will need to talk to system 3 to validate the message

Yes, this is why any post claiming to "roll their own security" is a farce. The current way security works (using HTTPS) is to pre-install trusted certificates on the machine at the hardware manufacturer or OS installation level. The keys that are pre-installed are this initial high level (usually root certificates), can then secure lower level machines, which can further secure even lower level machines. This only works because we trust our hardware manufacturers (e.g., Dell, HP, etc.) to install the correct certificates. These manufacturers are analogous to "System 3" in the example above.

I'm not really sure how to explain this any better, but let's try:

Our cast of characters: Service A, Person 1, and Person 2.

Person 1 makes an API request to Service A. It's done over HTTPS, and Person 1 trusts that the response received is genuine, because it came over HTTPS, and certs were verified and so forth.

Person 1 sends the text of the API response to Person 2. Let's say Person 2 is a journalist, and Person 1 is a potential source. Person 2 doesn't know Person 1, and so logically doesn't blindly trust them. But Person 2 wants to report on some factual thing about the API response. How can Person 2 be sure that Person 1 hasn't modified the API response before Person 2 receives it?

If the API response is signed somehow, Person 2 could go retrieve Service A's public key (directly from Service A) and independently validate that the API response was actually received from Service A, and that it hasn't been tampered with.

I'm not sure I agree with the original author that this is a critical problem that needs to be solved, but that's the gist of it.

> How can Person 2 be sure that Person 1 hasn't modified the API response before Person 2 receives it? ... If the API response is signed somehow, Person 2 could go retrieve Service A's public key (directly from Service A) and independently validate that the API response was actually received

The problem here is that Person 1 cannot be trusted, so they can't be the one providing the signature. Even if Person 1 signs, hashes, or encrypts the message when sending to Person 2, all it can prove is that it came from Person 1, not that it came from Service A.

If Person 2 wants to be sure that the message it got from Person 1 actually came from Service A, it will need the signature provided by Service A, not Person 1. To get the valid signature, it will need to contact Service A.

This is a long-winded way to say that if you want to trust what you got, you've got to go to the original source. HTTPS provides tools to do exactly this, and it's special because every time you buy a computer or server, trusted certificates are pre-installed on the machine. Without that initial level of trust, you can't trust anything you receive from anywhere.

They can share the exact same encrypted bytes they received. Others can verify those bytes with the cert as normal. Or do I have it wrong, and the traffic is only signed with the user signature after the handshake?

You have it wrong. I also thought that was the answer. See these responses to my question on StackExchange https://security.stackexchange.com/questions/193559/saving-a...

Looks like the message is encrypted with a key derived by both parties but this produces a symmetric encryption. The client then has the ability to spoof encrypted server communications.

SSL data is encrypted with a symmetric, per-session key. Only the initial handshake is done through asymmetric encryption, and only the initial handshake uses the global public/private key pair.

I appreciate the point the author is making, but why is this a problem of the api provider? And why would api providers be incentivised to implement the solutions to this problem?

I'm not sure they would be. They are the best party to sign/verify that responses are genuine, though.

Unfortunately in some (most?) cases you don't control list of trusted authorities on the endpoint.

That's not how I read the article. This bit from right near the beginning explains why he's interested in more than what https provides:

If I call https://api.example.com/?id=123 myself, then I can be reasonably sure that the data has come from the server unaltered.

But what if someone emails me a JSON file? Or someone re-hosts an XML dump? Or a hacker claims to have uncovered proof of...?

As data-journalism gathers pace, it's vital to be able to see whether data is authentic.

How do I know your comment is legit if you didn't sign it? What is special about APIs?

Sign all the things!

But for real.

I had an idea a while back: with all the photo/audio/video manipulation and ‘deep fakes’, there should be a certification available that accompanies (very expensive) cameras/recorders with protected firmware that signs each and every raw frame/recording, so the original is always available upon request from any type of inquiry or investigation.

Sounds like an invasion of privacy. Your personal pictures and videos shouldn’t have a unique digital signature so the police or a bad actor can identify which device it was taken on.

This seems like a good idea at first but what about whistleblowers or informants?

Wouldn't have to tie it back to you. It would be enough to say "Canon affirms this photo was taken directly on a model X". Which would then tell you that this was the unedited photo and could be trusted.

> Sounds like an invasion of privacy. Your personal pictures and videos shouldn’t have a unique digital signature so the police or a bad actor can identify which device it was taken on.

You've misunderstood the point. The goal is not to identify users. The goal is to sign messages to ensure they were actually outputted by a service.

You mean like how it's done with printers? https://www.eff.org/pages/list-printers-which-do-or-do-not-d...

This is not the solution for this essentially unsolvable problem. It offers a false sense of assurance since there will undoubtedly be ways of extracting the signature apparatus and then using it to produce fake signatures. It also creates a dangerous opportunity for all kinds of authoritarian actors to inject tracking and identifying information in digital artefacts.

It's much less painful to abandon the idea that photographs, videos and sound can be ultimately trusted.

Nikon had something like this (the "Image Authentication System", which used a builtin key to sign the images) but somebody managed to pull the signing key off the camera:

https://blog.elcomsoft.com/2011/04/nikon-image-authenticatio...

Same thing happened with Canon's implementation the year before.

Thanks!

I figured it had already been tried.

I wonder if it can still be done.

> "How do I know your comment is legit if you didn't sign it?"

The comment is fake. The points in it are valid, though.

The big difference here is whether a comment or API response is claiming any kind of authority over certain data, or is a self-contained argument where the validity doesn't depend on where it comes from.

Then this new technology might be useful

https://developers.google.com/web/updates/2018/11/signed-exc...

It's part of a family of technologies developed by Google to rehost AMP pages without losing the origin.

Digging themselves out of problems that didn't need to exist. They could have just penalised fat pages.

While I agree with your point, and think AMP should not need to exist, Signed HTTP exchange might actually be great for other types of very cheap, dumb caches or CDNs that only have to rehost one file to cache a page, and not even serve it through SSL.

I know I might use it as a web designer to send interactive snapshots of how the website currently looks/works to the client; instead of using an ephemeral URL and a tunnelling/proxy service.

Part of HTTPs negotiates a bunch of symmetric keys and these are used to encrypt and ensure the integrity of the data. Because the keys are symmetric it is possible for either party to fake a transcript of the conversation after the point that the symmetric keys are negotiated. So you cannot use HTTPs to prove to a third party a response or a request when using client certificates is genuine.

> So you cannot use HTTPs to prove to a third party a response or a request when using client certificates is genuine.

HTTPS is not sufficient for such third-party exchange, but they are necessary.

If you (say "System 1") ask for a twitter feed from an intermediary (say "System 2"), which in turn scrapes the data from twitter itself (say "System 3"), then HTTPS can indeed help. You can't rely on any hash or signature produced by System 2, because they could modify the data and make up a new hash. Instead, you would need to rely on the hash from System 3, to confirm that what you received from System 2 is authentic. Only tools like HTTPS, with their pre-installed security certificates, can do that.

I think there could be a case made based on the fact that the HTPPS connection isn't necessarily established by the service you want to connect to. Especially in the cloud world where TLS can be terminated on a load balancer or similar network service instead of the service you wanted to address. This assumes threats between these two services, similar to what the author describes as manipulation by intermediaries.

In this case the author criticizes manipulation of the content by third parties. I hope this is not another argument for a digital notary and believe that there are better technical solutions to that problem. Digital ledgers seem to be a niche some people try to establish.

In the end HTTPS is just transport encryption, not content encryption or content signing.

From an architecture point of view I think it is even advantageous if not every services handles HTTPS itself.

> APIs do not need to sign every response, they just need to properly use HTTPS.

That doesn't make sense. The problem described in the article was that the request documents aren't signed, thus there is no way to verify if a response document was actually outputted by a web service. HTTPS handles transport, but has no responsibility regarding request documents. If the problem we need to solve consists of ensuring if a given JSON was actually outputted as a response to a HTTP request to a service (say, if a Twitter message was actually sent by user X on day Y) then how would HTTPS help?

>> APIs do not need to sign every response, they just need to properly use HTTPS.

> That doesn't make sense. The problem described in the article was that the request documents aren't signed, thus there is no way to verify if a response document was actually outputted by a web service.

Suppose that you ("System 1") is getting data from an intermediary ("System 2") which in turn is getting data from Twitter itself ("System 3").

Even if the intermediary System 2 signs the data, all that proves is that you got it from System 2. It does not prove that it was not tampered with when System 2 got that data from twitter directly, System 3. It would be easy for System 2 to tamper with the data, and come up with it's own signature.

If you, as System 1, want to confirm that the message you got from the intermediary (System 2) is authentically from twitter itself (System 3), you would need to get the signature from twitter directly. You would need to ask for twitter's signature (System 3's signature), not the intermediary's.

So system 1 will need to contact System 3 in order to validate the information it got from System 2. There's no way around this.

Depend on your use case, HTTPS may not give you the right guarantees. For example, if you are trying to check an online license (where the client running the current application is essentially a potential adversary), HTTPS gives you no guarantees that the license server you think you are talking to is the real license server. You could pin the certificate to work around this, but then you have to reinvent revocation and renewal of the certificate for your own application. A longer lived, application specific signature may be a better option.

In other cases, HTTPS's end-to-end encryption may not be completely desirable - for example, apt does not use HTTPS as it would interfere with simple caching proxies, and it offers no extra assurances than a verified signature does.

> HTTPS gives you no guarantees that the license server you think you are talking to is the real license server

What? This is the entire purpose of HTTPS! To do this securely:

1 - you create your server;

2 - you tell a certificate authority (like Symantec, Digicert, or Let's Encrypt) how to find your server, and they do the diligence to make sure you're not trying to redirect another person's server;

3 - they provide you with a private encryption key and expose the public decryption key;

4 - you save the encryption key on your server.

Now, anyone who receives any information from your server can be sure that it came from you because (a) your server is the only one with the encryption key and (b) we trust Symantec, Digitcert, etc to distribute the public decryption key.

This might not be the most salient place to start when critiquing this article but it did irk me that they get the definition of idempotent wrong even while linking to Wikipedia.

>That is, the data returned from a query is always the same.

That is not correct. Only the system state remains the same. As stated in the linked wiki page:

>[...] the end result is always the same as the result of the initial execution, even if the response differs.

That's a fair criticism. What is the correct word for "this request always generates an identical response"?

Immutable

Aha! Yes. Thanks. I'll update the blog post.

Immuteble describes data, whereas an API call is more like a function call, in which case we'd call the property "referential transparency", which means, a function always returns the same result for a given input value.

"referential transparency" makes up half the definition of a "pure" function, with the other half being the absence of side effects.

Pure

Pure also means no side effects, which isn't, strictly speaking, what OP means :)

Deterministic

Cool concept but totally overblown need.

If the usage for api signing is to verify that leaked private messages are legit then I don't think we should have such tech in place.

If anyone has any more legitimate use cases that auth does not account for I would love to hear.

Here are a few that I considered including in the post...

The city's car park API told me I could park here - why have I been given a parking ticket?

The hospital's drug API told me I could mix these two tablets.

The bank's API told me a client hadn't received the funds I sent.

Basically, if I have to go to court and say "I have proof that your system told me XYZ."

Better yet:

Someone sends you a tweet with your address, a picture of your house and a picture of a gun, then deletes the tweet the next day.

Code signing does something similar; it allows data to be published on untrusted mirror sites. But it's not clear why most API owners should care about this use case?

In particular, Twitter allows users to delete their Tweets. If you think it's good to be able to take back what you said, signing tweets doesn't help.

The author seems to have written this article in a very web-centric way, as if "API" could only possibly refer to web service APIs that go over the open internet.

But of course sometimes (you might even say often) APIs don't go over a network at all and don't cross the sort of boundaries that require signing.

For example, the Linux kernel has an API that applications can use. There's no need for the response to open(2) to be signed. If you can't trust the result, you can't trust anything else that is happening either since you are at the mercy of the kernel.

The author primarily works with web APIs. But, for example, the Kernel's USB API may want to verify that it is communicating with a specific device. And, for audit purposes, log the signed input.

This sounds like coming full circle back to SOAP. Library support for signing an XML doc and adding the cert for offline verification is pretty well established.

The main problems with SOAP were practical problems - bad dev experience and inefficient, like almost everything XML.

But the goals of SOAP were not bad.

I hated it at first but once I realised the use-cases I changed my mind. When you have regulations telling you to keep hold of documents for 5+ years, and you want to digitise them, then communicating through signed XML documents is a pretty nice solution. The format is suitable.

I mean, we're not talking about Twitter searches or any such useless stuff. It's inter-bank transactions, direct debits, payment pre-authorisation, etc. Everything that you really do need a paper-trail for for years and years.

Author presumes that the platform (in charge of signing the response) finds the signature valuable. This presumption is faulty - indeed, plausible deniability is a feature, not a bug for these platforms. Which platform wants to give the government a loaded gun by cryptographically verifying that they hosted content or behavior that broke the law?

It might make sense to also add a section to the article considering why you'd want to verify an API signature instead of calling the API again.

Some examples I could think of:

- You don't have access to (that part of) the API, but to its public key

- The data in question has been deleted

- The entire API or even Service is gone

- The API call involves some complex calculations that takes a long time

- The data might have to be verified without access to the API or the network as a whole

- You are being Surveilled and/or your access to the API or some of its endpoints is being restricted

Isn't this solved by HTTP-Signatures, which is a draft RFC? I guess not, as there's no way to forward that signature to a third party without some imposter-ing, depending on the HTTP headers that were signed.

Finally a use case for blockchain. Not new though, look around at the "trust economy".

Basically this is the raison d’etrê of Verified Claims.

Not every API response is a verified claim.

Next?

and how do you suppose to get the end user the certificate? because if it's trough the same channel, none of this matter.

The idea (if I understand it correctly), is that you can get a signed API resopnse from anywhere, and the public key from the signer to verify the legitimacy of the data in the API response without having to call the API again.

Reposting the comment as top level, as I haven't seen it mentioned anywhere.

Google has developed Signed HTTP Exchanges https://developers.google.com/web/updates/2018/11/signed-exc... that can then be bundled https://web.dev/web-bundles/ .

Its main motivation is the rehosting of AMP pages while verifying that the cache hasn't tampered with them; but it could be extended to sign and bundle API calls for redistribution if needed.

Reply on news.ycombinator.com