Hacker News Reader

> The place where REST falls down, really, is transactions. For simple enough schemas and transactions involving only one relation, it's easy enough to have URIs for resources and collections. But for anything much more complex than that REST simply fails. Instead one ends up writing an end-point that accepts POSTs whose request bodies encode complex transactions, and this ends up resembling SOAP (basically RPC over HTTP) rather than REST.

It's perfectly RESTful; resources that represents actions are just as valid as any other resources. I feel like the mistake here extends a common mistake of object-oriented analysis where the valid idea that objects should correspond to nouns are somehow combined with forgetting that actions are nouns.

If you have an API with complex transactions (especially if you need to expose state of those transactions), why wouldn't they be resources? Why wouldn't you post a representation to a collection endpoint? How is that not exactly REST?

> It's perfectly RESTful; ...

Yeah, so this is where just what the meaning of REST is comes in. There's no representation of the "transaction" resource that you can GET, therefore they are not like the other resources (the ones you can HEAD/GET). You can only POST representations of transactions that internally contain ad-hoc encodings of CRUD operations on multiple other resources. This really feels like it should have been a first-class operation in HTTP rather than constructed in an ad-hoc way as needed.

> There's no representation of the "transaction" resource that you can GET

Except in the most trivial cases, it's probably essential that there be such a representation and that you 201 the transaction with a URL for the resulting resource. The nature of aggregate transactions is that you probably don't want to do a 200 and done, especially if you are supporting arbitrary combinations of CRUD actions as you suggest.

> You can only POST representations of transactions that internally contain ad-hoc encodings of CRUD operations on multiple other resources.

“Ad hoc”? No more than any other resource representation. Not that I think most applications that need a transaction resource of some kind need an arbitrary combination of lower level actions; most need well defined classes of domain transactions that are each more constrained.

If you design in this model, 'GET transaction-uri' should get, y'know, the payload of the transaction, or a status resource, or whichever you asked for based on proactive or reactive content negotiation, or whichever the server wishes to make available.

It's just such a mental departure from the last 20 years of 'do-the-verb-to-the-noun' object oriented design that it's not often done.

Sure, but it's still not like a resource, and you still have to write code to implement it. You could never have a user-agent construct a transaction because the construction is completely ad-hoc and will differ by application. It's just not generic.

> Sure, but it's still not like a resource

It is exactly a resource.

> and you still have to write code to implement it.

Well, someone, somewhere has to, just like any other resource.

> You could never have a user-agent construct a transaction because the construction is completely ad-hoc and will differ by application.

This is not true in any sense where the same is not true of lots of resources.

> It's just not generic.

While REST may favor using suitable generic representations where available, being generic isn't a requirement of REST.

REST is DRY, not “Don’t write code for anything.”

It seems like the GP is arguing that REST is insufficiently generic and you’re responding with “That’s not true, because REST isn’t generic”. Why is it desirable to write the same boilerplate code for every resource and now (presumably) every combination of resources that are operated on by every kind of transaction? Or are you making a different argument?

It's not clear what 'generic' means here, or in the parent, or anywhere in this thread. But in my view, this post [1] is saying that payloads that define a complex transformation or atomic manipulation of multiple independently-accessible resources require an extremely intelligent user-agent to construct case-by-case, because there's no widely-accepted terminology to describe arbitrary transformations of arbitrary data, or to introspect raw, semantically non-marked-up data ahead of time and figure out what needs to be done.

In other words, to construct a resource you could POST to '/bank-transfer' that removes x amount from account y and adds it to account z requires the client to know about what these things mean. But the point of this post [2] is that that's precisely true, and it shouldn't be surprising, because this is exactly how every resource works.

The disconnect between the expectations in this post [1] and this post [2] stems from the fact that most of these CRUD-over-HTTP APIs are actually implemented as thin middleware that parses info from the URI and hands it to database to retrieve. Resulting data items are then serialized using simple conventions as some commonly-used, non-hypermedia structural format, like JSON (no links, no rels), and served as 'application/json', devoid of any programmatic semantic meaning. A human reads this, and uses a combination of common sense, natural language skills, documentation, and trial-and-error to extract meaning from such a payload (e.g. "customerId" probably matches our customer number). For both retrieval and the inverse of this as storage, this is simple, and works: fields are matched by name, and the values are taken at face value.

To the author of this post [1], this probably feels 'generic'. It takes low effort to implement on the server, it can be templated to work on any resource it can find in the database, and assuming an intelligent (i.e. human) client, these resources are practically self-describing! But to the author of the other post [2], this approach seems like a hardcoding of a particular behavior into client based on the the structure of a resource (e.g. a Customer resource), which doesn't seem meaningfully different from the same burden that would be required to implement a resource that describes a domain verb rather than a domain noun (e.g. a Bank Transfer resource).

[1] https://news.ycombinator.com/item?id=18488976 [2] https://news.ycombinator.com/item?id=18489402

Generic refers to whether the existing user-agents can construct the request without the help of some JavaScript code. For example, your browsers know how to construct HEAD, GET, POST, and other requests, but they don't know how to construct a POST to a transaction end-point that encodes a bunch of POSTs on individual resources.

Assuming you accept things like links and forms, you can get a user-agent to collect input and ad-hoc "job" descriptions from the user and fill in the required structure in your specific application model. It's no more exotic than building up a shopping cart and submitting an order. Once you reify the process, you can add asynchronous monitoring and lifecycle controls as REST operations on these resources.

Don't mistake a custom resource model with server-side processing with a requirement for a custom user-agent. All of the above could be done with pure, form-driven interactions or augmented with client-side processing to provide a slightly nicer UX. Of course, that doesn't mean you will have a generic web service that can perform all different transactions.

I'm not sure what you're thinking of. What I've see is where a transaction resource is expressed as a collection of modified documents. E.g., in pseudo JSON:

  [ doc1, doc2, doc3 ]

Usually each doc consists of some metadata (at least the doc URI, probably a revision number of the base document, maybe more depending) and the modified document content.

(actually, it might be better like this: { "uri1": doc1, "uri2": doc2, "uri3": doc3 })

I'm not sure what your objection is, but this seems pretty generic to me.

So the client GETs documents. It modifies some, puts them in a collection to form the transaction and POSTs them back to the store.

The reason you can't POST a transaction [1] is primarily that due to ~25 years of Web inertia that has hardwired people's brains into thinking that the request lifecycle is that a request comes in, you deal with it, you emit your response, and then you drop all state on the floor. For a good chunk of those 25 years nothing else was possible due to memory constraints, but today things are bit more fluid, yet the old ideas persist, often still unexamined [2].

I could build you a transactional API in Erlang, Haskell, or Go fairly easily, where it is easy and not necessarily unidiomatic to have a thread persist for a long period of time. It would do the obvious thing: You POST to get a transaction ID, the thread starts a DB transaction [3]. You do your various other POSTs, passing the transaction ID. When you're done, you POST to the transaction itself the commit request and get the response back. If you take too long, the coordinating thread will automatically roll back. It's not terribly different from other REST-based workflows that exist in the world; a REST interface [4] is not particularly required to not have workflows in it. There are various technical issues, some of which I footnoted even, but it's conceptually perfectly possible.

There isn't a representation of any sort of resource until a developer creates it. There's no particular reason why a transaction can't be such a resource, any more than any number of other very abstract resources that exist in the world.

[1]: Properly speaking, not a GET because it changes the state of the world.

[2]: Unexamined != wrong, by the way. Some of the old issues still obtain. Some, however, do not. Dropping all state is still often a great idea, but there's a lot of APIs and code bent around a repeated cycle of "drop all state -> immediately reconstruct it all in the next request" that wastes a lot of developer and computer time for something that is not always necessary anymore if you're not scaling to the moon. Scaling code is great, but don't pay the price for 10^8 when you're never going to pass 10^3.

[3]: Possibly it could build up a list of desired changes you want, or a number of other ways around holding open DB transactions for long periods of time, but let's go ahead and do it the hard way for demonstration purposes.

[4]: At least, the modern colloquial sense of the term. I'm not particularly concerned about the details of The Thesis, which I consider a nice piece of work, but not something particularly binding on the rest of world, and not necessarily that much more interesting than an extended blog post since I do not believe it to have been based on anything like extensive real world experience or anything else that would give it more heft than any other thesis.

> There's no particular reason why a transaction can't be such a resource, any more than any number of other very abstract resources that exist in the world.

Well, true, but also kinda besides the point? I mean, what you are saying is essentially "there is no particular reason why a transaction can not be represented as a sequence of bytes". But by that measure any sort of protocol or transport that can move sequences of bytes is equally good/useful.

The fact that you can in principle tunnel anything over anything is not a useful insight when trying to determine the usefulness of a particular abstraction, at best it tells you that you can always add another layer of abstraction to add missing functionality.

> You POST to get a transaction ID, the thread starts a DB transaction [3].

I don't disagree with the logic here, but this is going to get old fast in a modern load-balanced, multi-region/zone topology, since it means pegging the client's state interaction to a particular backend process.

Sticky sessions are the classical answer to this, but they are very much something you want to avoid these days, because they require that every proxy along the path to your app is aware of the stickiness. If you end up deploying your app to, say, Google Kubernetes Engine, you don't even get a load balancer that supports stickiness.

The only way I would implement this would be with a level of indirection. When a request comes in, start the thread, open a listener socket (with, say, gRPC), then store the socket address in some reliable shared data store (Etcd, ZooKeeper, Postgres, etc.). Any requests from the same client can then be routed to that socket internally. If you're writing this in Erlang, you can replace "socket" with "process" and be done in half the time.

That might work okay for latency-sensitive OLTP stuff, but a good pattern I've used before is to model transactions as explicit state. The client uses CRUD actions to build up a "batch" of data (e.g. in a CMS-type app, create a folder, then create a document in the folder), which is stored separate from live data and completely invisible. Once the client is ready to commit, it sends a commit request, and the backend takes the stored state and tries to apply it to the main database.

There are some benefits to this asynchronous, batch-oriented view of transactions: For example, you get audit logging and debuggability "for free", and the backend can be smart about conflict merging and retrying without involving the client. It also allows the client to post incomplete data; it can create a document before the folder exists, even though the document at this point is not yet valid according to business rules, and later in the same transaction it can "patch" the document with the right parent folder. This maps well to the kinds of ETL that apps tends to want to do, where it's nice to write a thin dumb import process that doesn't have to keep lots of local state.

"Scaling code is great, but don't pay the price for 10^8 when you're never going to pass 10^3."

I am responsible for applications both where holding open a DB transaction as I describe would be suicidal levels of stupid as I expect hundreds of events per second as the applications hits maturity, and perfectly acceptable because the load of the application is measured in single-digit write sessions per week. (They're high-value sessions on a per-session basis, having massive effects, but there aren't a lot of them.) It's obviously a fail to engineer the 10^8 system to 10^-2 standards; it's less obviously a fail, but still a fail, to make the opposite error.

I was talking about the operational aspect, not scalability. Read my first paragraph again. :-)

I don't see any huge issues with the design in terms of scalability.

> The reason you can't POST a transaction [1] is primarily that due to ~25 years of Web inertia that has hardwired people's brains into thinking that the request lifecycle is that a request comes in, you deal with it, you emit your response, and then you drop all state on the floor.

I think that's still true if we're talking ephemeral state, like session data. Caching such data is an optimization, but conceptually we should be able to toss it at any point.

If you need to keep state, then it needs to be durable with a well-defined lifetime, and so this state should itself be a resource.

> If you need to keep state, then it needs to be durable with a well-defined lifetime

Why? Why can't you drop it at any time and have the client repeat the interaction on any kind of problem?

I'm not saying that it shouldn't be a resource either. Those things do not exclude each other.

I meant server side state. Client provided data isn't state from the server's perspective.

We are talking about server transactions. It's server-side data sent by a client.

Just FYI, there's a pending enhancement in PostgREST[1] that would allow us to take advantage of HTTP/2 multiplexing and have multiple requests executed atomically in a single transaction.

[1]:https://github.com/PostgREST/postgrest/issues/286

Neat!

Another thing, regarding eventing, is that you can use HTTP for that: just use chunked transfer encoding in never-ending responses. I've written a tailfhttpd that does this for tailing files a-la tail(1) -f, as you'd expect. You want to have some sort of application-level heartbeat though because HTTP/1.1 chunked transfer encoding sure doesn't have that, but otherwise you can absolutely sit there waiting for events that come along over HTTP, and this can be extremely fast (my tailfhttpd is C10K).

> The place where REST falls down, really, is transactions.

I don't agree. The standard way of supporting transactions with REST is to provide a transactions endpoint where clients start a transaction by POSTing a new transaction resource, POST their DTOs to build up a unit of work, and when they are done just commit the transaction. That's pretty much a one-size-fits-all solution as that's the standard way to implement transactions at the database level.

I was going to say the same thing and was pleased to see that someone beat me to it. If you combine unit of work pattern with continuation tokens then you can handle bulk transactions involving very large numbers of records with a simple rest interface.

Yes, and I've said as much. But the user-agents can't construct this without JS.

> Imagine an HTTP/1.1 verb named TRANSACT

Why not just use RPC at that point? If one of the concatenated messages has a failure, REST can only report back an error code for one of them. Also, the client can't really stop sending once it has started. The whole connection would have to be torn down to cancel out early. Also, transactions usually involve both read and writes. Doing a stream of writes that are not conditioned on some reads is often not that useful. (I.e. check if this account has money, and then transfer to some other account).

For one thing because the CRUD structure enables a bunch of things. For example, suppose you wanted to front a service with a reverse proxy that performs authorization: if the verbs and resources encode all you need to authorize supplicants, then you could do it generically, otherwise your proxy would have to understand the application's resources in much more detail.

Of course, you can do this with RPC as well. But HTTP is so... widely used, while RPC frameworks are a) a dime a dozen, b) none so widely used that you can find reverse proxies for them.

  > > Imagine an HTTP/1.1 verb named TRANSACT
  > 
  > [...]  If one of the concatenated messages has a failure, REST can only report back an error code for one of them.  [...]

REST as it is can report whatever you want because the encoding of transactions is ad-hoc, so you just have to make a provision for conveying all the error information that you want.

A TRANSACT verb does not exist, but if it did it could easily returns a status code, message, and error response body for each operation in the transaction.

> But HTTP is so... widely used, while RPC frameworks are a) a dime a dozen, b) none so widely used that you can find reverse proxies for them.

The beauty of gRPC is that it is HTTP. All proxies that support HTTP/2 implicitly support gRPC.

> REST as it is can report whatever you want because the encoding of transactions is ad-hoc, so you just have to make a provision for conveying all the error information that you want.

But at that point it isn't really REST. The single request-single response semantics are lost. The error codes don't map cleanly to the operations. In some cases they do, but not generally.

> The beauty of gRPC is that it is HTTP. All proxies that support HTTP/2 implicitly support gRPC.

s/gRPC/SOAP/g

And what were the complaints about SOAP? Right, that using POST for everything was not RESTful, which among other things means things you could have gotten with GET can't be cached by middleboxes, and so on.

  > The beauty of gRPC is that it is HTTP. All proxies that support HTTP/2 implicitly support gRPC.
  > 
  > But at that point it isn't really REST. The single request-single response semantics are lost. The error codes don't map cleanly to the operations. In some cases they do, but not generally.

No, you'd still have that: you'd be doing more than one request/response in a linked way.

> And what were the complaints about SOAP? Right, that using POST for everything was not RESTful, which among other things means things you could have gotten with GET can't be cached by middleboxes, and so on.

Honestly that wasn't much of an issue. If you're doing something transactional then the GET parts still need to participate in that transaction and so you don't actually want to be caching them. The big problems with SOAP were XML and especially XML schema, the fact that there was no good way to have a single source of truth about what your data model looked like, and especially the corresponding absence of clarity about what kind of changes to the data model were forward compatible.

And even with all that, SOAP applied intelligently (rather than autogenerated by one-size-fits-all tools) was often nicer to work with than REST is. Part of me thinks the only reason REST has a better reputation is that developers are forced to hand-craft REST interfaces whether you want to or not, whereas with SOAP it's very easy to have a tool generate an API that's crappy but "good enough".

In my mind, SOAP is terrible because of all of the tools that implement the standard but do not interoperate. The fact that a .NET and a Perl SOAP implementation can't communicate without a very specific configuration implies that the standard isn't clear enough on the points that actually matter.

The other issue I have with SOAP is really an issue with "generic RPC mechanisms" in general. In trying to generically solve every single possible RPC situation with the same tool, they end up creating an opaque tool with a ton of configuration options that expose the entire problem domain back to the user. So if that's necessary, then why bother trying to hide the details of the solution? Just let the developer decide how best to create the RPC abstraction in his or her own application.

> REST for the control plane works because there are few enough "resources" that you can have a URI for each and CRUD works.

Isn't the issue also that entities on the control plane often have relatively simple state machines? You can reduce REST complexity by designing APIs that manipulate the state machine rather than thinking in terms of a pure CRUD model.

For instance, the need to hold client-side state disappears in transaction models where you request a desired state and then have the option to wait for the actual state to correspond or just go away and check later to see how it's doing. The poll request (if there is one) translates from looking for transaction completion to waiting for the desired state.

Edit: for clarity

I referred to schema simplicity elsewhere in that comment, which is not the same thing, but similar enough to state machine simplicity.

Combining multiple changes in one request is actually part of the oData standard (which is REST): http://docs.oasis-open.org/odata/odata/v4.0/errata02/os/comp...

You can send multiple retrieves and changes in one multipart request.

Would you go through a concrete example?

Sure. I'll use PostgREST, which has a RESTful data plane interface to PostgreSQL. There's no UI as such, though you can build one with, e.g., Admin-on-REST. There's only trivial transaction support, even though there's some non-trivial query support. All transactions consist of a POST to update one row or insert many rows all on the same table, or a DELETE to delete rows. All non-triviality has to be done via functions or views with INSTEAD OF triggers.

But now imagine you could construct a transaction like this using curl as an example:

  $ curl -X DELETE https://pgrest.foo.example/sometable/somerowid -- -X PUT https://pgrest.foo.example/sometable/someotherrowid -d @file_containing_row_data -- ...

If you use one curl(1) invocation per operation then you get no transactional capabilities. But you can't do it in one request because HTTP itself has no transactional capabilities.

Instead you have to construct a single request whose body encodes all those requests in some PostgREST-specific manner (as opposed to some generic and standard manner that all applications could use).