Notes on streaming large API responses | Hacker News Reader

Notes on streaming large API responses

simonwillison.net

131 pointssimonw52 comments

There are at least two other reasons I prefer small (paginated) requests.

First, it’s much simpler to define meaningful latency SLOs. If requests take roughly the same time, I can say that requests (or even requests “of this type”) should complete with p99 under 300ms. If they’re very variable, it’s harder to know when the service is degraded. (You can use time-to-first-byte, which is usually more uniform, but you can miss degradation that affects the data stream. Then it’s tempting to look at throughput, but that’s impacted by client speed, which you usually want to ignore here.)

The other is that bulk requests can make it much cheaper to DoS the service. If all requests take about the same amount of work on the server, then forcing the server to do work requires proportional client resources to make the requests. If you can make one small request that causes the server to go off and do a bunch of work while the client does almost no work, that’s a cheap way to DoS a server.

Either do efficient pagination, or convert into an already fairly well solved problem, like large file downloads, via batch exports. Batch exports just move the problem in house though.

Either way, don't rely on a continuous long running connection. You need to be able to restart or retry without starting over. That's table stakes, and it makes a resumable pure streaming API look a lot like a paginated one. The hard bit of pagination is efficient resuming, not limiting to a page size.

Efficient pagination means having enough information on each subsequent request to pick up reliably and efficiently from where you left off, e.g. using a sequentially allocated indexed ID on the root table (to which other data is joined).

Sometimes the database needs convincing to use the right index of you have joins, as DBs are generally tuned for extracting a handful of rows at a time. I've had more reliable success (over a multi-year maintenance timeline) by doing a single paginated query over a root table and doing separate queries to get the joined data, using a page of root IDs at a time for the joins. Doing the join in code also gives leeway for parallelization of the data load across cores (MySQL queries are single threaded), or even across replicas.

This really annoys me in PostgreSQL. Django encourages you to use joins across one-to-one relationships, but the query planner doesn’t know how to apply the limit before the join. It’s relatively simple to change select_related to prefetch_related on all paginated queries, but it still feels like the query planner should do it, especially for left joins (inner joins are slightly more complex to check).

I described an implementation using keyset pagination in the article, which sounds like what you're advocating for here.

Indeed.

I tried this once. The problem I ran in to was that you really only see the benefit of the streaming if all of your components do it properly, including the database, server, client, and application logic that consumes the data. Streaming is really easy to screw up and hard to debug if it's not working. For example at one point my problem was the client's JSON parser. I had carefully chosen a streaming parser, but it turned out that the streaming had to be explicitly enabled - by default it slurped all the data.

Wouldn't it help with slow clients?

If I can send a bunch of small chunks to the client and the first few can already be processed, it doesn't matter that I had to load all of the chunks at once from my database.

This is indeed a fun and challenging type of API endpoint to implement.

I built a streaming statistics server that ingests data from edge nodes and aggregates the data into a memory database.

The server has an API interface and one of the endpoints will stream all the data in either a compact or pretty JSON format.

example of the datasets: millions of unique unordered keys with each key containing a list of time series data.

The memory overhead to stream Gigabytes of data is limited to a couple of MB (buffers for encoding and writing).

I ended up having to write my own JSON encoder to achieve the small memory overhead, JSON formatting, and write performance.

This is how I do it: store the large data files on GCS. Then, offer them via an F1 Node.js endpoint but make the process 'smart' and offer them in 10MB chunks. The client sees a chunk progress (3/10 etc) and a chunk % progress (0 - 100 for each chunk). The client also sequentially persists the large text responses as they arrive

This has saved both my very limited F1 server from sad OOM exceptions and my iOS vm'd client from crashing while processing too much text

I do the same for uploads

If you want to get real fancy you can have your F1 server generate a self-signed GCS url and have it returned as a 307 redirect. Even less load!

I need to serve it from an endpoint because the response includes multiple text files from GCS bundled together (as many as can fit a 10MB chunk). But yeah otherwise that would work (for downloads)

Do you just read the response linearly? What made you choose GCS instead of something like memcache With some kind of chunking?

So the great thing about using an F1 instance is that it completely falls under the free tier (28hrs/day)

There is no memcache in node.js F1 but what I do is I use the GCS container itself as stream memory instead of the usual node.js memory multer (yes, this is possible!)

This only adds a few ms to the read op which is entirely fine

28 hrs/ day...?

1 instance running all day = 24 hrs

Times as many instances as you have. So 28 hrs = 1 instance running all day + an extra 4 hours of a second instance

It auto scales

Thanks, thought I had slipped into an alternative dimension. Appreciate you helping me take the edge off.

No probs mate

For a project where I worked on we offered streaming TSV responses and always finished the response with a last row containing a comment: `# Finished downloading.`. Since we also managed one of the client packages to access the endpoints, we check for that when the download is finished and raise an error if it is not there.

In a multipart message, a missing (final) boundary delimiter is a clear indication of partial content. Returning one might might work for CSV files.

This is fun, just recently I had to implement a large data export feature. So I looked into generating zip file streams with this package: https://github.com/arjan-s/python-zipstream

You can generate a zip stream and then stream it over http.

That's really cool, I hadn't seen that library.

Is that better than just using GZIP though?

You can send the user multiple files on one download this way.

I am switching all our endpoints from paging to streaming.

Currently for the paging endpoints some of the clients are opening hundreds of connections to slurp tens of gigabytes of data as fast as possible.

Switching these to streaming directly from database improved the performance of a single request about 100 times as clients can use 10 times less connections and still fetch whatever they need 10 times faster than before.

As to drawbacks, the biggest one is additional effort needed to educate people on proper streaming, design of APIs, implementation and correct use of Reactor (ReactiveX implementation for Java).

There are couple of technical drawbacks.

Resuming a streaming response is a little bit more involved. My ideal solution, whenever I can, is to produce the report in order of some database index and if the call fails, have the client submit the ID of last received response when retrying the call.

Achieving optimum performance requires a bit of thought and planning. A lot of internal processes require individual things to be batched together (we try to never direct calls to database for single items, opting to process everything in batches of at least 100 or 1000).

When the endpoint has functionality that needs to be executed multiple times concurrently to get needed performance, sometimes an internal function also has its own concurrency because it needs to run something else multiple times in parallel to achieve its target internally. This can quickly cause the developer to loose track of how much stuff is running concurrently. The solution to this is that rather than components returning Flux<Something> typically they return Flux<Flux<Something>>. This can be thought as function saying "my result can be fetched from a collection of multiple substreams of results, and you can decide however many you are comfortable running concurrently". This allows to give the calling function the control of concurrency at the cost of little bit of mental yoga.

What about the other drawbacks mentioned in the article, like connection draining and error messaging?

Well, connection draining is no harder than with normal connections.

I think of this problem following way: you already have to design your system to be able to handle network faults and node crashes. So just kill the node and rely on existing solution to handle the error.

This also gives the opportunity to test your fault handling logic.

Error messaging is a little bit more complicated.

At first sight, in normal request-response you get either successful response or an error.

But if you look a little bit closer, in fact you get 200 response header followed by response data, which may itself fail to be transmitted. So technically you already need to be able to resolve an error that happens after you have received 200 header.

In case of streaming response the difference is not in how the response is being sent to the client but how the client is processing it.

The client has option to wait for entire response (header followed by entire stream of data) and only start processing the response after entire response has been received. This is no different from a normal, non-streamed request response.

The client also has option to start processing the response while it is being received. The response does not need to be streamed from the server nor it needs any kind of streaming protocol. I have been stream-processing XML responses two decades ago with no special streaming support.

One solution I use is that is that the client architecture treats this data as speculative meaning that an error can undo entire batch received.

Another solution is that the design is such that the data makes sense even partially received, the processing of response is idempotent and the client will be asking, ad infinitum, to receive the rest of the response until the call succeeds. This of course has its own reliability problems (what if the error is permanent?)

I‘d like to mention that streaming large results is now trivial in the Java world as well, thanks to reactive spring (spring webflux). It even works with SQL thanks to r2dbc (reactive SQL driver API).

It‘s quite refreshing to develop with reactive spring.

Don’t mention reactive spring on hacker news, everyone here writes in GO and javascript.

They’d rather re-invent the wheel every 5 minutes than use any out of the box java solution.

How about returning large API results as NDJSON (newline-delimited JSON)?

Modern browsers can read the response of a network request in a streaming fashion

See e.g. https://canjs.com/doc/can-ndjson-stream.html

We are using ndjson. It is pretty good for our case of processing couple of terabytes of 1kB-20kB documents daily.

Some of the calls return millions of objects. In such cases we opt to return very small amount of information (ideally just ID) and have the client batch responses and direct batches of IDs to another endpoint to fetch full documents, with response also streamed using ndjson.

But there is nothing wrong with returning entire 20GB of data except that the client gets no other option than wait for entire set being returned through a single connection.

I mentioned that here - I agree, it's a really simple and useful format for streaming:

> Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON.

It does have the same minor disadvantage as CSV in that it can be truncated (by an error or a dropped connection) in a way that can look like a successful download.

that was my first thought too. ndjson makes a lot more sense there. You don't want to keep the entire streamed API response in memory anyway.

I don't think you use a large response when there are other alternatives. One problem is see is with tests. You can expect that there are some problems that might arise with very large data sets, but it may take some computer time to set-up and break-down such tests.

I've found writing tests against these endpoints to be pretty easy - I populate the associated database table with one record more than the default page size and then run a test that retrieves the full result and checks that the expected records are present. No need to run tests that generate 100MB responses every time - though I will do a few manual tests of large responses to check that RAM usage is as-expected.

Here's my test in Datasette for the streaming CSV response - I already have a table with 1001 items in it setup for another test: https://github.com/simonw/datasette/blob/0.57.1/tests/test_c...

I followed the exporting to S3 approach. Unfortunately it takes a while until those files become available, even though the AWS API says they are. We had many issues with this approach in the past. Better host the response yourself....

As of just over a year ago they now provide strong read after write consistency which should solve this problem.

Just ran into the problem again last week. Even though we verify the files availability using the boto3 client...

Once the initial S3 PUT request succeeds, the object is guaranteed to be available immediately AFAIK.

As long as you haven’t tried to GET the object before the PUT succeeded. Otherwise there can be some eventual consistency for the negative cache to be overwritten.

This can be quite annoying if you have e.g. a separate process polling for an S3 object. The object can appear in a HEAD and then return a 404 on the subsequent GET.

It sometimes takes minutes for the file to become available. Meanwhile we get a 401 unauthorized. Even though we have something in place like that:

  def is_file_uploaded(file_name, request_id):
      for i in range(S3_UPLOAD_ATTEMPTS_COUNT):
          try:
              s3.Object(BUCKET, file_name).load()
              return True
          except botocore.exceptions.ClientError as e:
              logger.warning('File still not uploaded to S3. 
  Check - {}/{} (request_id - {})'.format(
                  i+1, S3_UPLOAD_ATTEMPTS_COUNT, request_id
              ))
      return False

As mentioned in the sibling comment, this has been fixed by AWS a few months ago and now all S3 operations are always strongly consistent (not eventually consistent)

Just happened again last week. What's even worse the location from where we tried to access the file also had impact. Like we were able to reproduce the issue from one location, but not from another.

That's interesting - have you tried contacting support? I don't think this is supposed to happen.

You might also have better luck using the region-specific S3 endpoint rather than the global domain name.

That's a fair point. I'll switch over to region specific! Thanks for putting me in that direction.

And no - haven't contacted support so far. Will assign someone from my team to get in touch with them.

You're right, I encountered this issue as well a few years ago. However, this has changed recently. S3 now offers strong consistency https://aws.amazon.com/s3/consistency/

Nope it doesn't :(

Ah, good old eventual consistency.

I used to work at a financial data provider. Our clients were institutional investors (e.g. banks, mutual funds). We would FTP our data directly to our client's machine (or SFTP for our more savvy clients).

Indeed, FTP makes the fintech world go round. Banks do a lot of things in batch (e.g. clearing, settlement) and FTP matches that nicely. At least in North America, this is starting to be disrupted a bit as faster payment instruments and transfer options arise.

> If you’re streaming a response, you start with an HTTP 200 code... but then what happens if an error occurs half

Maybe a 202 Accepted code makes more sense

200 looks perfectly valid. Any HTTP response can be interrupted at any time by any error down the stack (app crashing, server, VM, network etc). 200 does not a guarantee complete delivery.

It does not. 202 means the request has been accepted and will be processed somewhere somehow. Here the response is being returned immediately, just streamed.

Reply on news.ycombinator.com