Strategic Pagination Patterns for APIs

roxeem.com

3 points·roxeem··5 comments

We have to deal with this at work. It’s the pain of moving large data across the network via HTTP. You have no idea how large the data is going to be or how long the SQL query will take to execute u til after the HTTP request is issued by which time the request TTL is already defined.

This is a self inflicted pain though. You shouldn’t be using HTTP to move large data. A streaming socket is so much more efficient.

The transport protocol doesn't matter. You will have the same problem no matter the transport layer. Yes socket is probably going to be faster, but not paginating correctly will cost you more memory, CPU and money.

That is so far outside of reality. How do you think your VOIP calls work? They don't paginate data. They just pipe from one buffer to another where the second buffer is a network socket. The network socket doesn't care what the data looks like or how big it is. It will just push forward what its fed until either end of message or socket termination.

What you're referring to is two completely different requirements. VOIP or video streaming don't need database queries. Those data are sequential by nature. You don't need to page them. Also even those types of data are transported in chunks.

The transmission mechanism, the socket, doesn’t care what data traverses it. Whether the incoming data is a stream or chunks is up to how the receiving end wishes to intercept the data until message completion. The receiver can store incoming packets as an array of chunks or pipe the incoming into a buffer. The former allows for more convenient segment inspection while the later allows for immediate execution on message fragments.

Reply on news.ycombinator.com