Show HN: Rendering One Million Datapoints with D3 and WebGL | Hacker News Reader

Show HN: Rendering One Million Datapoints with D3 and WebGL

blog.scottlogic.com

200 pointsColinEberhardt11 comments

Very cool! I’ve done visualizations on similar size datasets by mapping tsne output to fake latitude/longitude coordinates and rendering it with mapbox gl. It feels like a hack, but then you get all of the panning/zooming interaction, styling, and rendering labels with collision avoidance for free. For smaller datasets I pass in geojson directly, but for larger ones I precompute vector tiles so you can lazy-load more of the data when you zoom in. There are a lot of things built for mapping that are relevant for other domains, I wish that there was a common base that could be extracted that mapping libraries adapt to geographic data but data visualizations could adapt to non-geographic data.

I’ll have to play around with your technique more, encouraging to see you get good performance eagerly loading a million data points!

Deck.GL has some similar examples. I think the reason this makes sense is that geo data viz had to pioneer huge datasets - which are less common in other fields.

I can recommend LargeVis[0][1] which I used to layout and Polymaps[2] to visualize tiles of the https://wikiscape.org article point cloud.

[0] https://github.com/lferry007/LargeVis [1] https://github.com/elbamos/largeVis [2] http://polymaps.org/

Scrap polymaps, use leaflet instead: https://leafletjs.com

Nice write up Colin!

fwiw quadtrees don't make much sense when you're showing a tooltip for ~1 million points: https://roadtolarissa.com/scan-sorted/

That's really interesting, thanks for sharing those results. So easy to go down the route of premature optimisation!

Scott Logic is probably the highest-quality company technology blog I've encountered in a while. I keep returning to it for WebAssembly and WebGL topics and it is consistently high quality. I'm not even sure what Scott Logic does, but they sure know their stuff :)

Kudos, Colin and team.

Their head office is round the corner from my office in Newcastle, UK, and I occasionally bump in to their devs in the local pub and at the JS meetup. They're very good at what they do and lovely people too.

https://www.scottlogic.com/what-we-do/

;)

Thanks Paul, really appreciate that :-)

Just FYI, we're a medium sized software consultancy based in the UK. We do quite a lot of work in financial services, which is why you'll find quite a few posts on performance and cutting-edge web technology. I started the company blog 12 years ago and now we have ~50 authors creating some amazing content. Unfortunately I don't have the time to write quite as much as I used to!

The scatterplot of the library data is fascinating — looks like an organism.

Reply on news.ycombinator.com