Hacker News Reader

Comment by detaro | Hacker News Reader

Somehow I feel like WebGL in the browser isn't going to just handle it for things like the main image on the project page, where the compressed dataset is > 1 GB already.

View on HN

Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. You can even have it anti aliased practically for free.

> Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer.

1GB to the browser for rendering... Sure, it's doable. So is eating 1KG of wings for lunch. Doable, but very far from being a good idea.

Once again, there is no need for very much to be in memory at one time. This is accumulation and buffers can be reused. I'm not sure how much more I can simplify this other than to say that addition doesn't care about ordering, you can just add continually without needing access to any of the other data that has been rasterized.

I think this was already said above, but it still seems to be getting confused, so to repeat: Datashader renders everything out of core, in the server. So it doesn't matter whether a client could successfully accumulate results for a large data incrementally; to use WebGL directly one still has to send all of the data to the client eventually. With Datashader the dataset is never sent to the client in the first place; it stays on the server, which could be a remote HPC system with thousands of cores processing petabytes. Datashader renders the data into an image-shaped array on the server, then sends that (much smaller) array to the client, so that the client never sees any data larger than the available screen resolution. This is no claim that doing so is unprecedented or some crazy new idea, just that Datashader lets you render datasets regardless of their size, completely independently of any client (browser) limitations, and without having to serialize the data over an internet connection.

Reply on news.ycombinator.com