10 Years of Open-Source Visualization: Did I learn anything from D3.js?

D3.js is, to me, one of those tools that makes things possible rather than making things easy (although I only use javascript once every year or two so it's always a slog). This is big: it's like a zero-to-one step, rather than incremental progress.

This fall I made an interactive visualization of the fine-grained plate tectonic model I have built for the Himalaya and Tibet for a research project[0]. The model building process (i.e., finding the relative motions of the plates) is largely solving a system of linear equations, so it's not immediately apparent what the results really mean and whether they are acceptably realistic.

D3.js allows me to explore and understand my own scientific results so much more thoroughly than the static quiver plots that I was making in Matplotlib--being able to pan and zoom a globe (not just a projected map!) and see the tectonic plates move through time is phenomenal. Being able to share the results so trivially as a web page is a huge force multiplier as well.

Thanks so much, Mike!

[0]: http://rocksandwater.net/blog/2020/11/tibet-blocks-js/

I love D3.js

Here is my favorite example - try click the 4 buttons at the top:

https://archive.nytimes.com/www.nytimes.com/interactive/2012...

A little D3 war story: I wrote a solver for the Schrödinger equation, which would render a wavefunction with D3 and SVG - cool right.

It was very slow to render. Eventually I found out why: D3 was round-tripping the SVG through text! The render time was dominated by the browser parsing the thousands of 15+ digit floating point numbers.

I added a call to round the floats to two decimal places, so they would be textually shorter, and it was a huge speedup. What a strange optimization.

I later rewrote it using WebGL; here it is should you care: https://ridiculousfish.com/wavefiz/

I love D3 and am always shocked by Mike Bostock’s willingness to help. Ten years of D3, yes, but ten years also of Mike answering questions in great detail across GitHub, SO, Twitter, etc. And that’s on top of maintenance + running a startup. Some people just have unbelievable drive.

In other news, I’ve enjoyed using D3 over the years, and always jump on visualization projects as a result. Here’s my favorite thing I’ve made while I was learning D3: https://www.jwilber.me/permutationtest/

I found this so useful. I'm building a suite of open source tools at https://datasette.io/ and the section "Teaching is the most impactful aspect of tool building" was something of a light bulb for me.

"To maximize your impact, then, teaching must be central to your strategy. This means documentation, tutorials, examples, videos, tweets, and more."

I have pretty comprehensive documentation ( https://docs.datasette.io/ ) but almost nothing in the way of tutorials - and I added the first demo video to the website just last week.

This has given me a ton to think about. Thanks, Mike!

> Of all forms of documentation, examples seem to be the most effective.

This is applicable in almost all types software, not just graphical ones like D3. Many software lack this and can be made much better/attract more adoption if there are more examples.

I love this post. I can't tell you how many times I've started trying to use d3 and gave up after a couple hours because "all I wanted was a bar graph" (at least 3). Learning it is still on my bucket list because it is truly an amazing library.

I think this is great:

> The only way to recognize — and then bridge — the gulf of understanding between you and people using your tool is to observe their struggles and talk to them. It’s amazing how quickly glaring flaws are revealed this way. Answering questions on Stack Overflow (or GitHub, Twitter, Slack, or wherever else) is not a selfless act of altruism; it’s a chance to learn, to find where people struggle and hear their perspective. Each question is an opportunity to help one individual, but that question can also inspire a tutorial, an example, or even a feature that prevents others from hitting the same issue.

What I love about being a front-end engineer is seeing how people use what I build, which almost always brings up "obvious" issues and improvements that weren't so obvious when you're working on it yourself.

Now that I'm mostly building developer tooling, this gets even better: not only are the users like me so I can empathise with them better, but I know where they're going with their questions and observe them "in the wild", and easily figure out what help they need without needing to set up specific UX research sessions.

But this primarily gets fun if you use it as a way to scale up your support. Which means it's important to keep the following in mind:

> But there’s also a limit. As one person, you can’t help everyone. And you can’t make everyone happy because your tool is only a tiny part of their life. Don’t make that your goal: focus on learning and broadening your perspective. If it stops being constructive for you, then stop. Period. And don’t feel bad!

Don't know a damn thing about visualization or how to use d3.js but I have a shitload of respect for Bostock's work on it, and his role in using it at the NYT.

Love this window into his thought process.

I'm also happy to see someone as accomplished as him say that randos on the internet are also capable of hurting his feelings. It makes me feel substantially less alone in that regard.

I owe a lot of my career to people like Mike Bostock who have made open-source libraries like d3 free and available to the world. I'm sure I'm just one of countless developers who feel the same like me. Eternally grateful

In some ways I miss my time as an R programmer. Back then I could type R and make various sorts of rough useful plots of data quickly. 10 years of nonstop programming later, I no longer use R, and I cannot make rough useful plots quickly in either Python or JS. (There is nothing quick about activating the right virtualenv, starting a jupyter server, getting a notebook running, and trying to remember how to import and use matplotlib. And JS is worse).

> To maximize your impact, then, teaching must be central to your strategy. This means documentation, tutorials, examples, videos, tweets, and more.

I think this is one of the main reasons the FastAPI framework is very successful. The documentation teaches you about everything and doesn't assume any prerequisites besides general Python knowledge. I've used it recently, and it's such a delight to work with.

D3.js is cool - and you can do more with it than just data visualisation https://datacrayon.com/posts/visualisation/visualisation-wit...

This might be slightly off-topic, but I can't seem to find a direct solution.

I'm looking to build a complex dependency/timeline graph for a web of interconnected stories in a media franchise. Each story has a release date where it needs to appear, and an arbitrary list of dependencies for previous episodes/stories/interludes which would have a line drawn to that dependency. The end result would be a sort of hybrid digraph/timeline, enabling a reader to get the 'minimum path' to catch up to a certain story in the collection.

D3 looked promising, but there's no direct equivalent I can see in the examples. For those that are experienced with D3, is that sort of thing achievable? Should I be looking at D3 or something else?

I used d3.js for one of my first programming projects. I was grateful and impressed that almost every question on Stack Overflow was answered by Mike Bostock or Jason Davies.

I'm (a little) older and (a little) wiser, and know this kind of support is a huge amount of work. It sounds like "focus[ing] on learning and broadening your perspective" is what fueled all those answers I so appreciated.

I like D3 and Observable! So well done experience for the users!