Zoomable, animated scatterplots in the browser that scales over a billion points | Hacker News Reader

Zoomable, animated scatterplots in the browser that scales over a billion points

github.com

148 pointssamwillis16 comments

I'm pretty sure I have seen this or something very similar somewhere before but I can't remember where. It is pretty impressive.

The background map they use though for this is definitely wishful thinking in my area. I clicked on points that looked to be near me and discovered that the census data used for those points has people living on a couple of roads that haven't been built and won't be built for at least two years if things go like they have on the other road construction projects within a few miles of me. Today the roads only exist on planning documents posted by the city.

Highcharts has had a "boost" module for a long time (at least back since 2015/2016) that switched the rendering over to WebGL for large data sets, but I don't think it ever got up to 1B points.

https://www.highcharts.com/docs/advanced-chart-features/boos...

Is the name a reference to the democracy game from Scientific American back in the 80ies (or so)?

Also, is there a good equivalent for node-edge graphs?

Ideally, you'd embed the graph into 2 or 3d first, then visualize it as a scatterplot.

Visualizing the edges at scale doesnt yield nice results in general. It's simply too noisy.

The way to do it is to reduce the graph to some 300d or 500d embeddings, then use TSNE/UMAP/PACMAP to reduce that to 3d. Then visualize.

My prefered way is to use some first order embedding method like GGVec in this library [1] (disclaimer I wrote it). Node2Vec and ProNE don't yield great embeddings for visualization (the first is too filamented, the second too close to the unit ball).

Another great library to do this work is GRAPE [2]. Try first-order embedding methods, or short walks on second order methods to avoid the embeddings being too filamented by long random walk sampling.

[1] https://github.com/VHRanger/nodevectors

[2] https://github.com/AnacletoLAB/grape/

Thanks!

I don't suppose you have a link to that game? It sounds interesting and I'd love to learn more.

https://wiki.c2.com/?NomicGame has a bunch of information around it, but the original article is behind a paywall (https://www.jstor.org/stable/24966607).

Reminds me of a more ambitious project where you can view and zoom 1000s of images: <https://gitlab.com/galapix/galapix>

There is also a simpler Rust imitation of that project: <https://github.com/google/pix-image-viewer>

Demos:

<https://www.youtube.com/watch?v=OJz0dz3oG-Q>

<https://www.youtube.com/watch?v=9-Mg6MkE-tg>

    Github issues, even low quality ones, are welcom here.

This a refreshing line to read. I wonder if the missing “e” (welcom vs welcome) is intentional. It sounds like and easy (low quality?) PR.

I do find it funny when you have this glaringly bad bug that effects everyone and the bug template wants you to report everything down to your DNA. Sometimes it is nice to just report: latest released version, here's my stack trace...

Nomic are also the ones behind gpt4all. Such a powerhouse team. https://github.com/nomic-ai/gpt4all

Great tool to add to the visualization toolbox! It does seem to lack the visual fidelity at large scales that it resovles at finer ones. I wonder if it could extend to the render + raster approach of Datashaper, while still running everything in browser?

Nice. At least 7 years ago I saw a similar demo: https://demo-taxis.heavy.ai/ for an example.

Is the difference that nomic's billion points are rendered entirely client-side?

yes, it's using webGPU to render the whole thing

It seems I don't have the necessary graphics extensions available on my mobile device (mainstream device using mainstream browser). Is this supposed to work on mobile?

nice, https://github.com/laurolins/nanocube is/was another interesting approach to the many points issue, though it's seen no updates since 21 afaik

Reply on news.ycombinator.com