Modern browsers can read the response of a network request in a streaming fashion
Modern browsers can read the response of a network request in a streaming fashion
Some of the calls return millions of objects. In such cases we opt to return very small amount of information (ideally just ID) and have the client batch responses and direct batches of IDs to another endpoint to fetch full documents, with response also streamed using ndjson.
But there is nothing wrong with returning entire 20GB of data except that the client gets no other option than wait for entire set being returned through a single connection.
> Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON.
It does have the same minor disadvantage as CSV in that it can be truncated (by an error or a dropped connection) in a way that can look like a successful download.