Post-REST
tbray.org
tbray.org
The data plane is not at all like that. There can be many many resources, and very complex transactions.
PostgREST, a REST interface to PostgreSQL, shows you can have REST for the data plane, but it achieves that by forcing one to have simple transactions with any and all complexity wrapped into PG functions so that the REST interface can remain simple.
The place where REST falls down, really, is transactions. For simple enough schemas and transactions involving only one relation, it's easy enough to have URIs for resources and collections. But for anything much more complex than that REST simply fails. Instead one ends up writing an end-point that accepts POSTs whose request bodies encode complex transactions, and this ends up resembling SOAP (basically RPC over HTTP) rather than REST.
Imagine an HTTP/1.1 verb named TRANSACT whose body consists of a concatenation of the requests that you really want to do, and with a header controlling / reflecting transaction control options (e.g., atomicity). If HTTP had such a thing, then it would be a lot easier to build a proper RESTful interface to the data plane. But HTTP has no such thing, and probably never will.
Incidentally, it's useful to consider how NFSv3 had RPCs for every little thing, while NFSv4 has only one RPC (COMPOUND) that is... a lot like the TRANSACT I imagine above -- it's just not that crazy a thing to imagine. The transaction complexity problem in HTTP is not unique to HTTP!
It's perfectly RESTful; resources that represents actions are just as valid as any other resources. I feel like the mistake here extends a common mistake of object-oriented analysis where the valid idea that objects should correspond to nouns are somehow combined with forgetting that actions are nouns.
If you have an API with complex transactions (especially if you need to expose state of those transactions), why wouldn't they be resources? Why wouldn't you post a representation to a collection endpoint? How is that not exactly REST?
Yeah, so this is where just what the meaning of REST is comes in. There's no representation of the "transaction" resource that you can GET, therefore they are not like the other resources (the ones you can HEAD/GET). You can only POST representations of transactions that internally contain ad-hoc encodings of CRUD operations on multiple other resources. This really feels like it should have been a first-class operation in HTTP rather than constructed in an ad-hoc way as needed.
Except in the most trivial cases, it's probably essential that there be such a representation and that you 201 the transaction with a URL for the resulting resource. The nature of aggregate transactions is that you probably don't want to do a 200 and done, especially if you are supporting arbitrary combinations of CRUD actions as you suggest.
> You can only POST representations of transactions that internally contain ad-hoc encodings of CRUD operations on multiple other resources.
“Ad hoc”? No more than any other resource representation. Not that I think most applications that need a transaction resource of some kind need an arbitrary combination of lower level actions; most need well defined classes of domain transactions that are each more constrained.
It's just such a mental departure from the last 20 years of 'do-the-verb-to-the-noun' object oriented design that it's not often done.
It is exactly a resource.
> and you still have to write code to implement it.
Well, someone, somewhere has to, just like any other resource.
> You could never have a user-agent construct a transaction because the construction is completely ad-hoc and will differ by application.
This is not true in any sense where the same is not true of lots of resources.
> It's just not generic.
While REST may favor using suitable generic representations where available, being generic isn't a requirement of REST.
REST is DRY, not “Don’t write code for anything.”
In other words, to construct a resource you could POST to '/bank-transfer' that removes x amount from account y and adds it to account z requires the client to know about what these things mean. But the point of this post [2] is that that's precisely true, and it shouldn't be surprising, because this is exactly how every resource works.
The disconnect between the expectations in this post [1] and this post [2] stems from the fact that most of these CRUD-over-HTTP APIs are actually implemented as thin middleware that parses info from the URI and hands it to database to retrieve. Resulting data items are then serialized using simple conventions as some commonly-used, non-hypermedia structural format, like JSON (no links, no rels), and served as 'application/json', devoid of any programmatic semantic meaning. A human reads this, and uses a combination of common sense, natural language skills, documentation, and trial-and-error to extract meaning from such a payload (e.g. "customerId" probably matches our customer number). For both retrieval and the inverse of this as storage, this is simple, and works: fields are matched by name, and the values are taken at face value.
To the author of this post [1], this probably feels 'generic'. It takes low effort to implement on the server, it can be templated to work on any resource it can find in the database, and assuming an intelligent (i.e. human) client, these resources are practically self-describing! But to the author of the other post [2], this approach seems like a hardcoding of a particular behavior into client based on the the structure of a resource (e.g. a Customer resource), which doesn't seem meaningfully different from the same burden that would be required to implement a resource that describes a domain verb rather than a domain noun (e.g. a Bank Transfer resource).
[1] https://news.ycombinator.com/item?id=18488976 [2] https://news.ycombinator.com/item?id=18489402
Don't mistake a custom resource model with server-side processing with a requirement for a custom user-agent. All of the above could be done with pure, form-driven interactions or augmented with client-side processing to provide a slightly nicer UX. Of course, that doesn't mean you will have a generic web service that can perform all different transactions.
[ doc1, doc2, doc3 ]
(actually, it might be better like this: { "uri1": doc1, "uri2": doc2, "uri3": doc3 })
I'm not sure what your objection is, but this seems pretty generic to me.
So the client GETs documents. It modifies some, puts them in a collection to form the transaction and POSTs them back to the store.
I could build you a transactional API in Erlang, Haskell, or Go fairly easily, where it is easy and not necessarily unidiomatic to have a thread persist for a long period of time. It would do the obvious thing: You POST to get a transaction ID, the thread starts a DB transaction [3]. You do your various other POSTs, passing the transaction ID. When you're done, you POST to the transaction itself the commit request and get the response back. If you take too long, the coordinating thread will automatically roll back. It's not terribly different from other REST-based workflows that exist in the world; a REST interface [4] is not particularly required to not have workflows in it. There are various technical issues, some of which I footnoted even, but it's conceptually perfectly possible.
There isn't a representation of any sort of resource until a developer creates it. There's no particular reason why a transaction can't be such a resource, any more than any number of other very abstract resources that exist in the world.
[1]: Properly speaking, not a GET because it changes the state of the world.
[2]: Unexamined != wrong, by the way. Some of the old issues still obtain. Some, however, do not. Dropping all state is still often a great idea, but there's a lot of APIs and code bent around a repeated cycle of "drop all state -> immediately reconstruct it all in the next request" that wastes a lot of developer and computer time for something that is not always necessary anymore if you're not scaling to the moon. Scaling code is great, but don't pay the price for 10^8 when you're never going to pass 10^3.
[3]: Possibly it could build up a list of desired changes you want, or a number of other ways around holding open DB transactions for long periods of time, but let's go ahead and do it the hard way for demonstration purposes.
[4]: At least, the modern colloquial sense of the term. I'm not particularly concerned about the details of The Thesis, which I consider a nice piece of work, but not something particularly binding on the rest of world, and not necessarily that much more interesting than an extended blog post since I do not believe it to have been based on anything like extensive real world experience or anything else that would give it more heft than any other thesis.
Well, true, but also kinda besides the point? I mean, what you are saying is essentially "there is no particular reason why a transaction can not be represented as a sequence of bytes". But by that measure any sort of protocol or transport that can move sequences of bytes is equally good/useful.
The fact that you can in principle tunnel anything over anything is not a useful insight when trying to determine the usefulness of a particular abstraction, at best it tells you that you can always add another layer of abstraction to add missing functionality.
I don't disagree with the logic here, but this is going to get old fast in a modern load-balanced, multi-region/zone topology, since it means pegging the client's state interaction to a particular backend process.
Sticky sessions are the classical answer to this, but they are very much something you want to avoid these days, because they require that every proxy along the path to your app is aware of the stickiness. If you end up deploying your app to, say, Google Kubernetes Engine, you don't even get a load balancer that supports stickiness.
The only way I would implement this would be with a level of indirection. When a request comes in, start the thread, open a listener socket (with, say, gRPC), then store the socket address in some reliable shared data store (Etcd, ZooKeeper, Postgres, etc.). Any requests from the same client can then be routed to that socket internally. If you're writing this in Erlang, you can replace "socket" with "process" and be done in half the time.
That might work okay for latency-sensitive OLTP stuff, but a good pattern I've used before is to model transactions as explicit state. The client uses CRUD actions to build up a "batch" of data (e.g. in a CMS-type app, create a folder, then create a document in the folder), which is stored separate from live data and completely invisible. Once the client is ready to commit, it sends a commit request, and the backend takes the stored state and tries to apply it to the main database.
There are some benefits to this asynchronous, batch-oriented view of transactions: For example, you get audit logging and debuggability "for free", and the backend can be smart about conflict merging and retrying without involving the client. It also allows the client to post incomplete data; it can create a document before the folder exists, even though the document at this point is not yet valid according to business rules, and later in the same transaction it can "patch" the document with the right parent folder. This maps well to the kinds of ETL that apps tends to want to do, where it's nice to write a thin dumb import process that doesn't have to keep lots of local state.
I am responsible for applications both where holding open a DB transaction as I describe would be suicidal levels of stupid as I expect hundreds of events per second as the applications hits maturity, and perfectly acceptable because the load of the application is measured in single-digit write sessions per week. (They're high-value sessions on a per-session basis, having massive effects, but there aren't a lot of them.) It's obviously a fail to engineer the 10^8 system to 10^-2 standards; it's less obviously a fail, but still a fail, to make the opposite error.
I don't see any huge issues with the design in terms of scalability.
I think that's still true if we're talking ephemeral state, like session data. Caching such data is an optimization, but conceptually we should be able to toss it at any point.
If you need to keep state, then it needs to be durable with a well-defined lifetime, and so this state should itself be a resource.
Why? Why can't you drop it at any time and have the client repeat the interaction on any kind of problem?
I'm not saying that it shouldn't be a resource either. Those things do not exclude each other.
I don't agree. The standard way of supporting transactions with REST is to provide a transactions endpoint where clients start a transaction by POSTing a new transaction resource, POST their DTOs to build up a unit of work, and when they are done just commit the transaction. That's pretty much a one-size-fits-all solution as that's the standard way to implement transactions at the database level.
Why not just use RPC at that point? If one of the concatenated messages has a failure, REST can only report back an error code for one of them. Also, the client can't really stop sending once it has started. The whole connection would have to be torn down to cancel out early. Also, transactions usually involve both read and writes. Doing a stream of writes that are not conditioned on some reads is often not that useful. (I.e. check if this account has money, and then transfer to some other account).
Of course, you can do this with RPC as well. But HTTP is so... widely used, while RPC frameworks are a) a dime a dozen, b) none so widely used that you can find reverse proxies for them.
> > Imagine an HTTP/1.1 verb named TRANSACT
>
> [...] If one of the concatenated messages has a failure, REST can only report back an error code for one of them. [...]
A TRANSACT verb does not exist, but if it did it could easily returns a status code, message, and error response body for each operation in the transaction.
The beauty of gRPC is that it is HTTP. All proxies that support HTTP/2 implicitly support gRPC.
> REST as it is can report whatever you want because the encoding of transactions is ad-hoc, so you just have to make a provision for conveying all the error information that you want.
But at that point it isn't really REST. The single request-single response semantics are lost. The error codes don't map cleanly to the operations. In some cases they do, but not generally.
s/gRPC/SOAP/g
And what were the complaints about SOAP? Right, that using POST for everything was not RESTful, which among other things means things you could have gotten with GET can't be cached by middleboxes, and so on.
> The beauty of gRPC is that it is HTTP. All proxies that support HTTP/2 implicitly support gRPC.
>
> But at that point it isn't really REST. The single request-single response semantics are lost. The error codes don't map cleanly to the operations. In some cases they do, but not generally.
Honestly that wasn't much of an issue. If you're doing something transactional then the GET parts still need to participate in that transaction and so you don't actually want to be caching them. The big problems with SOAP were XML and especially XML schema, the fact that there was no good way to have a single source of truth about what your data model looked like, and especially the corresponding absence of clarity about what kind of changes to the data model were forward compatible.
And even with all that, SOAP applied intelligently (rather than autogenerated by one-size-fits-all tools) was often nicer to work with than REST is. Part of me thinks the only reason REST has a better reputation is that developers are forced to hand-craft REST interfaces whether you want to or not, whereas with SOAP it's very easy to have a tool generate an API that's crappy but "good enough".
The other issue I have with SOAP is really an issue with "generic RPC mechanisms" in general. In trying to generically solve every single possible RPC situation with the same tool, they end up creating an opaque tool with a ton of configuration options that expose the entire problem domain back to the user. So if that's necessary, then why bother trying to hide the details of the solution? Just let the developer decide how best to create the RPC abstraction in his or her own application.
Isn't the issue also that entities on the control plane often have relatively simple state machines? You can reduce REST complexity by designing APIs that manipulate the state machine rather than thinking in terms of a pure CRUD model.
For instance, the need to hold client-side state disappears in transaction models where you request a desired state and then have the option to wait for the actual state to correspond or just go away and check later to see how it's doing. The poll request (if there is one) translates from looking for transaction completion to waiting for the desired state.
Edit: for clarity
You can send multiple retrieves and changes in one multipart request.
But now imagine you could construct a transaction like this using curl as an example:
$ curl -X DELETE https://pgrest.foo.example/sometable/somerowid -- -X PUT https://pgrest.foo.example/sometable/someotherrowid -d @file_containing_row_data -- ...
Instead you have to construct a single request whose body encodes all those requests in some PostgREST-specific manner (as opposed to some generic and standard manner that all applications could use).
For example, with GRPC your client bindings are automatically compiled and you just use them.
Is it using protocol buffers? Do you care?
To a certain extent we do care. A lot of these things "just work" out of the box but there are some practical real world implications here.
At Datastreamer we push about about 100GB of JSON to our customers per day. This is all news and blog content.
Sometimes our customers complain that using JSON is a waste. Realistically though it costs them about $150 extra per month to parse the JSON vs using protocol buffers.
We made this decision because implementing JSON is a LOT easier than implementing protocol buffers.
Most of our customers end up putting a junior or intermediate engineer on the implementation and they're already familiar with JSON.
However, if the tooling hides the encoding and even the wire protocol this means we can upgrade faster.
We're still not at a point yet where the tooling is free. You tend to get stuck around something like GRPC or GraphQL anyway.
Hopefully this is resolve in the future and we can get all this infrastructure optimization for free.
This is what happened with SOAP because SOAP is too complicated to use without tooling. When it works, it's great. When it doesn't work, it's a nightmare.
The reason we use things like JSON and REST is because of the transparency and simplicity that it provides. There isn't much mystery when you need to debug. That's the trade off against things like bandwidth and performance.
I'm not really disagreeing with your point; it's easily to look at all this and wonder why we don't do it better and faster.
This is exactly why "RPC" solutions have always failed in the past.
The abstraction is too leaky. They quickly become almost impossible to debug, because the underlying implementation is needlessly opaque and complex (cough SOAP).
For the foreseeable future, there will always be more and better tools for analyzing and debugging HTTP and making ad hoc HTTP requests.
I don't, but other people do. For anything low latency, the cost of encoding/decoding starts to become extremely difficult to avoid.
But that said, I'm pretty sure GRPC can use JSON internally, so it's up to you which encoding to use.
Whether this is an argument against JSON and in favor of Protobuf or vice-versa depends on which language you're operating in. Most benchmarks I've seen show that Protobuf is faster to encode and decode for Java and Golang while JSON is faster for Python and Javascript.
IMO, if you're worried about low latency, you're also likely using a compiled language where Protobuf will be faster anyway.
Consider this: At the scale Google operates, fractions of a % of improvement are worth doing for saving the company (tens/hundreds/more) thousands of dollars in CPU/energy/anything, and they have the people to throw at the problem.
Does an alternative solution really improve any/all of throughput/latency/reliability?
"Doctor, it hurts when I do this." So don't do that; many HTTP libraries come with connection pooling / persistence built in, so it's quite common to not even need to handle this yourself. Should we also stop using relational databases because setting up a connection takes time?
As for the rest of it, sure, you might have a queue, or some orchestration layer. Is that your consumer-facing API? Just b/c I push a message into some queue implementation does not mean I want to expose that implementation to my caller.
"As for the rest of it, sure, you might have a queue, or some orchestration layer. Is that your consumer-facing API? Just b/c I push a message into some queue implementation does not mean I want to expose that implementation to my caller."
REST for external APIs, and queues for internal, more tightly coupled micro services seems like a good way to architect things.
That section is really stupid. He goes on about HTTP/2 and QUIC while seemingly being ignorant of HTTP/1.1's existence.
Keep-alive has existed for decades. Use it, it solves the problem definitively without adding any new ones.
(People have yet to figure out what the point of HTTP/2 and QUIC was. Except for Google's not-invented-here syndrome, of course.)
I actually built a websockets library (more proof of concept) [1] that uses a rest like syntax, implementing something similar to an express API. It exposes both a websockets server and http. It's FAR from anywhere near complete and just a concept. I have no doubt there are better solutions out there, but I just don't think REST is all that bad.
But I do also love @burtonator's idea of maybe not using JSON, but hiding the complexity of something like RPC.
The transaction itself can be a resource
This seems almost maybe an argument against taking microservices too far until a single service can't own the whole transaction
People think of resources as a table or row with CRUD operations rather than in the wider sense as an object with a generic set of operations to trigger transitions to a new state.
It's worth reiterating that REST operations are conceptually much more generic than CRUD. Nothing says that what you GET or PUT/POST need correspond directly to attributes of an object or columns in a database - in the widest sense they simply represent the current state of a resource and a state transition you wish to invoke respectively.
The point is not to (only) encode persistent state, but any state. In a sense representing state that is not persisted on the server is more essential - resources that are persistent on the server can be operated on without always sending the full state. But information that is not persisted on the server needs to be encoded and transmitted with the request/response in some way.
Why are we doing data schema type erasure at the Application layer? The "flexibility" and "encapsulation" aren't worth the manual hell of hand coding it back in all the way to the front end.
Why not expose SQL schemas all the way to the front end? Even if the Application layer makes a view that the data store doesn't know about, everything downstream could benefit through automation.
From there you get into issues mostly having to do with APIs that can be paginated, and how to optimize those connections to balance caching with network latency.
A somewhat related question that's fun to think about is why we don't put databases directly in our clients very often (especially SPAs). The ClojureScript community has been doing some good work exploring what it's like to put your client state in a client DB type of structure. There was also a compilation of SQLite to JavaScript done, I wonder how much usage that has?
If your API is SQL against some tables, you're promising you won't make any breaking changes to that -- or else you're promising you can roll out an update to all the servers and clients at the same time.
SQL against tables has a very large surface area, and there are many details that are different depending on the version of your database software. So, you'll find you can't update your database software without also updating (and testing) all of your clients. The more complex and more widely distributed your software is, the more time-consuming and expensive it will be to update anything. If you have clients outside your direct control and running on the same release cycle, you'll find you can't update your database software at all without breaking your clients.
So you only offer an ANSI SQL compatible parser, not specific to any database (and it would then be compatible / compile-able to postgres, sqlite, whatever), and you only expose specific views into the database.
Once you create such a public view, you treat it as immutable, and all API changes require new views. Each old view must function forever.
This is not actually all that different from REST apis today; we version them, old versions must exist forever (or all clients must be updated), you can't update your server software that serves the rest API without being darn sure it's backwards compatible...
The only real difference is we encode our CRUD operations in a non-sql language right now (ad-hoc json schemas usually) vs encoding them as the ANSI subset of SQL against some fictional database views.
All of your complaints apply equally well to REST apis I think.
It’s a much larger commitment than you need to be making for many, many APIs.
Because providing a SQL interface to all of your external consumers is a good way to quickly bring your service to its knees.
Maybe I'm missing something, but isn't that exactly what you can expect from a GraphQL API as a sweet spot between SQL and REST? Semantics of GraphQL enable a schema-dependent set of queries, not arbitrary queries.
> Anyhow, I see GraphQL as a convenience feature designed to make synchronous APIs run more efficiently.
For asynchronicity, GraphQL has subscriptions.
That being said, I agree GraphQL (of today at least) won't be a solution to every problem and I bet transports such as HTTP/2 and HTTP/3 will cause new things to be built which don't look like just a faster REST, GraphQL or MQTT. I would mention event sourcing and persistent logs in addition to just stream/queue processing.
Tooling (like API management gateways and API consoles) is not still all there for GraphQL and MQTT either.
I mean, in SQL useful queries are generally schema-dependent. Sure, you could have XPath/JSONPath/whatever extensions that allow you to query something closer to free-form documents, but your queries will still be schema-dependent in some way.
Does anyone have an opinion if it's a good idea and likely to gain adoption?
It sounds to me like it implements the sort of thing being discussed in this post
Is there a reason not to use REST then? I didn't quite understand.
Yes, it's mostly fine for that. Not always ideal, sure.
> let's say, to poll a resource every second and know if a worker has finished
No, you generally shouldn't poll every second. You should POST a subscription and then the other end should POST you a notification.
If you are in a situation where you need to poll instead of subscribing and waiting, then, maybe use HTTP long polling, maybe poll less frequently than every second, maybe use something other than HTTP, but in any case you are in a suboptimal circumstance.
(Of course, whether you should use HTTP as a transport protocol is orthogonal to whether or not you should use a REST architectural style.)
(That's assuming HTTP, which btw is not the only protocol which can implement REST constraints)
Cannot say much about REST constraints outside of HTTP so excuse my ignorance on that. But I think it was clear the parent was referring to HTTP, or most possibly, GET and POST HTTP calls returning JSON.
Rather than retry every second, you retry on timeout.
Transactional CRUD (post a form, submit an email, add to cart) is pretty well adopted and standardized as HTTP, so probably use that (even backend to backend, if the contract fits CRUD resource semantics).
RPC (non-CRUD interactions) or streaming updates (e.g. a spinner in a app representing a pending job) or low latency (chat messages / presence status, etc.) don't really fit to HTTP requests so great (requiring e.g. polling) so you should look at alternatives (probably something proxied over a websocket in the case of a browser front-end app).
I have seen huge benefits, performance being one, communicating using ZMQ (at the cost of complexity and reliability). A worker, running on your computer, consuming data fed through a socket from a broker, running on your server.
There can be. For example in the world of satellite internet where every byte is precious and every tcp connection is a massive performance penalty (3 way handshake * distance to satellite at speed of light = huge latencies), a REST API can be very wasteful. Something like GraphQL is much more suited to that case where you can get only exactly what you need, nothing more, and on a single connection.
GET is specified as not having a body and conformant HTTP implementations ignore the body of a GET, so in addition to violating RESTs general proscription on violating the underlying protocol (HTTP, in this example), you break compatibility with standard toolchain components if you do this.
What you seem to want is SEARCH pulled out from under the weight of WebDAV, i.e.: https://tools.ietf.org/html/draft-snell-search-method-01
GraphQL and SPAs are unconstrained tech solutions with questionable upside. Only well funded organizations can afford to throw money at projects that are more about attracting engineers than improving the bottom line.
You don't need a huge marketing campaign and a non-profit institution to drive adoption of technology that has a compelling value proposition.
- QUIC. Yes, it's UDP, so its connections don't behave the way TCP ones do, but it's still based on IP and UDP connections, so it still breaks down under a number of circumstances (rotating backends, mobile clients changing connection types, firewalls/proxies, etc)
- HTTP-based. If you use HTTP, you must accept its warts, like being stateless. Even considering a transport protocol that can multiplex connections, you still need to track all these individual stateless calls. Perhaps a protocol that took care of state might give you some benefits (such as workflowing?)
- Workflow orchestration is just distributed centralized state machines. It shouldn't have much to do with a replacement for REST, because the API isn't necessarily controlling a state machine, and the API (in this case HTTP) isn't stateful anyway...
Why don't we evolve APIs into a sort of language? Why make 4 separate requests (PUT this, GET that, munge it, POST a result, GET to confirm) when you know exactly what you want to do? We can instead build a language or spec where we can shove a variety of instructions (even a "workflow") into a request, let the backend handle the method of accomplishing it for us, and retrieve the result. This can be extended throughout microservice architectures.
The transport protocol is a separate issue. So you want lower latency, and you want less round trips, and you want to survive network interrupt. Design a transport protocol for that (QUIC is a good base). If you want it to survive even more interruption and allow clients to split networks, start looking at intelligent tunnels with advanced routing mechanisms and spread them along the path. But that's a lot more funky than simply replacing TCP.
Firewall support is a big concern but the other points you mentioned are design goals for QUIC — see section 9 of https://datatracker.ietf.org/doc/draft-ietf-quic-transport/?... (“Connection Migration”):
> The use of a connection ID allows connections to survive changes to endpoint addresses (that is, IP address and/or port), such as those caused by an endpoint migrating to a new network. … Not all changes of peer address are intentional migrations. The peer could experience NAT rebinding: a change of address due to a middlebox, usually a NAT, allocating a new outgoing port or even a new outgoing IP address for a flow.
Being stateless is one of the best things about HTTP, I wouldn't consider it a wart at all.
State should be very, very isolated. Ideally, stored in a database somewhere.
"Why make 4 separate requests (PUT this, GET that, munge it, POST a result, GET to confirm) when you know exactly what you want to do?"
At that point, you are implementing a database with transactions. So expose it over a SQL interface and be done with it (see Cockroach DB). This is solving a very different problem from most REST APIs. Exposing the ability for external customers to run arbitrary transactions against your servers is very dangerous in terms of provisioning, latency, and availability requirements.
Unless your business is providing a database as a service.
And no, you don't need a database and transactions just to perform four operations and return a result. Not every transaction will be affected by a change of state in the middle of the transaction. Sometimes transactions are duplicated, or replace a result entirely, and the latest one can win. A transaction in progress can also provide data to another transaction later, and the first transaction can be paused or terminated. We don't see complex interactions like this because the databases and protocols we use have strict separations between their parts, but we don't have to do things this way. The more functionality you surrender to a database, the less flexibility you have in the future, because now your big database is the bottleneck for operations.
We need more design paradigms that allow flexible operations, and protocols that can work with such operations to manage transactions by also working with state. I say we "need" this because it seems the state of the art of these systems has languished under proclamations about what we can and can't do, due to assumptions about how we must go about them, and what technology is readily available for use. But we're talking about a new protocol, so we don't have to carry the baggage of those assumptions with us.
The actual request was for a composable language, with 4 operations as one example. But a language isn't limited to just four operations.
By the way, this exists in the FHIR spec as a transaction Bundle:
https://www.hl7.org/fhir/bundle-transaction.json.html
But that, of course, now means you are implementing transactions. Which is one of the hardest things to implement in a distributed environment in a correct, scalable, and performant manner. Frankly, I would be very hesitant to trust such an implementation, if it wasn't the core competency of the developers who wrote it.
(Or unless they are just calling something else that implements the transactions, but at that point again you are just a very thin layer on top of a database.)
[1]: https://lightbus.org
Directly exposing your queue to external consumers seems very dangerous.
Having a well defined and well documented REST API for external consumers, and using queues for communicating between tightly coupled services seems like a much better solution.
It's ridiculous to think that anything can cover all use cases.
> The next stage of request handling is implemented by services that read the queue and either route an answer back to the original requester or passes it on to another service stage.
In this scenario, what are the mechanisms by which a service could route an answer back to the original requester?
I feel there's still too much competition/innovation in the message queue space though. Too many choices, with maddeningly different characteristics.
And seemingly you can tweak most of those to behave somewhat similarly between them, which makes judgement even harder.
The question is: can you use HTTP to transport event notifications?
Actually, can you use TCP to transport event notifications?
Yes, you can, but it's a bit unwieldy because you need some sort of keepalive / heartbeat with which to distinguish inactivity from transient network problems (and then attempt to reconnect). Then again, even if you were using UDP you'd have to do some sort of keepalive / heartbeat.
I use an HTTP server that supports "tail -f" via chunked transfer-encoding responses to indeterminate byte range requests (Range: bytes=${offset}-) that don't end until the file is removed, renamed away, or replaced (renamed over). This server also supports weak ETags constructed from (device ID, inode number, generation number), If-Match, and If-None-Match.
This allows clients to GET a resource whose body is the event stream and hang in there waiting for events.
Again, you need to add a heartbeat in the event stream as HTTP chunked transfer-encoding does not have this, and you need to detect network problems. But otherwise this works fantastically well and it is HTTP and RESTful.
Interesting, to me Kafka seems like the clear winner and the obvious choice for all of my queuing needs.
What am I missing out on from other queuing systems?