Why We're Building Observable
observablehq.com
observablehq.com
On top, it is forkable for others to build upon. It support comments, its reactive. When I explain it to people I say its like a combination excel + vscode + google docs + github. There is nothing like it!
I been trying to see how far I can push it for non- visualization software. Turns out... pretty far....
Firebase + Stripe => https://observablehq.com/embed/@tomlarkworthy/saas-tutorial
On Demand Minecraft Servers => https://observablehq.com/@tomlarkworthy/minecraft-servers
Zero deploy serverless cells => https://observablehq.com/@tomlarkworthy/serverside-cells
With serverless cells I can serve HTTP endpoints from a notebook! E.g. custom HTML like:
https://serversidecells-ibyw6dtm4q-ey.a.run.app/notebooks/@t... is generated from a notebook!
I am so productive in Observable because there is no context switch between development, using the product and debugging, it's all just one thing. Amazing, utterly amazing. I hope the Observable team can accommodate my use cases, as its clearly not what they had in mind.
I hope at some point they can offer some sort of offline support. The focus on ease of sharing is great, and the volume and breadth of content has really helped me to get more proficient with d3. The inability to open and edit notebooks offline/locally is a real inconvenience, however.
I have a small personal site on Netlify (with Next.js) and going forward I'm looking at hosting the sort of stuff I have on Observable there, simply as it's easier to work to when offline. The quality of the community and resources is a really valuable thing, but the need to be online is a real drawback at times.
Here is a project that we built last year (by its nature, it had a limited shelf life so it made no sense to have long term cloud support.)
If you inspect the source in the browser, you will see that the notebooks are hosted on the same server.
P.S. If you want online interactive notebook on your own site, I guess that is their business model either as a viral sharing of free notebooks or paid private team sharing. (I was on a team plan during the development of the project. But once the project is done, there is no need for us to continue on team plan and be interactive. I am glad that they offer support of offline production notebook.)
P.P.S. If Observable Team is listening, here is a business idea for them: observable playground as a service. It would be less a technical challenge than a business development challenge.
I also want to shout out Mike Bostock, one of the company founders (and creator of D3). I emailed him randomly to ask for some help with a d3 package and he replied the next morning.
Busy creators who nevertheless still make themselves available to engaging with the community always impress me!
One of the improvements on the computational notebook concept is that it seems to compute a data flow graph so it knows what cells to update when the data changes, without you needing to re-run cells in a particular sequence. The cells can also be out of order.
The editor interface is pretty slick, and I wish it were open source so I could use it in other projects.
(Mike Bostock, one of the founders, is also the creator of D3.js and the information visualization gallery bl.ocks.org. Here's my favorite: https://bl.ocks.org/mbostock/31bd072e50eaf550d79e)
Another great advantage they have is the data map on the right-hand margin. It's a good way of seeing how complex a notebook is getting as well as being a really efficient way of navigating through the data flow.
Also having learned a lot of D3 by reading through Observable tutorials, I'm curious what you've seen that doesn't seem to work when you port it to JS?
One thing I'm aware of is that you have to move a lot of the plotting code into the .then() method of promise when you're loading data.
------
[1] Observable's not JavaScript: https://observablehq.com/@observablehq/observables-not-javas...
The presentation is one more layer to understand between "here's the code" and "here's what the code produces". I do generally like the notebook format for exploratory data work, but it's not at all ideal as part of a workflow for writing code that will fit into a larger application, in my experience.
Observable is for JavaScript ecosystem. I wouldn't count on them implementing anything for Python.
It's different to something like Observable in that you import it into your existing Python environment (Colab, Jupyter, Airflow, etc.), build the reports from your assets, and push them up -- as opposed to having to do the analysis itself in our interface.
Yes I'm a fan; just would like to be able to write JS to visualize.
- Collaborative no-setup notebooks: different images with the most popular libraries pre-installed. See each other's cursors, very useful for troubleshooting, pair programming.
- Long-running notebooks: GPU ready. You can schedule notebooks right from the notebook view so you don't context switch. You can close your browser, disconnect from the internet, or shut your laptop and still see the notebook's execution.
- Automatic params/metrics/model tracking: no boilerplate.
- One click deployment: clicking a button deploys a model and gives you a nice "REST" API endpoint you can invoke.
If you're doing anything related to ML and use notebooks, you probably had the above problems. We're building around Jupyter notebooks because in our experience, our machine learning projects were not held back for lack of slick stylesheets.
- [0]: https://iko.ai
It is a bit silly to downvote this for not being true, when it is factual.
The Observable runtime is open source, and trivial to integrate into your existing react/whatever webapp.
There is also a vscode plugin that allows for live editing and rendering. It doesn't give you the ObservableHQ interface, but tbh I like working with files more. It also allows me to git everything.
https://marketplace.visualstudio.com/items?itemName=GordonSm...
You can't self-host the same way you can self-host the Jupyter Notebook or Jupyter Lab environment.
Or has that changed? Can I work on something offline without communicating with their servers?
You can self host observble, but you can't self host observablehq.
Literate programming just means the combination of code and documentation/visualisation.
Observablehq provides a nice inline code editor, but that's incidental to what observable is.
Git can also be used with a GUI but that GUI is not essential to what git is.
Simularly observable notebooks can just as well be created without the GUI, in a quite convenient way.
Knuths original literate programming environment consisted of a command line tool to split files that mixed pascal and tex into the pascal and tex source, to be compiled separately.
Which means that according to your definition, the original literate programming system, isn't literate programming.
All the best.
Hope you'll look at it again in the future. In a sense it's your loss, so good luck and take care.
It's like saying git (observable) can't be self hosted because github (observablehq) is not open source.
But observable (the reactive notebook technology) can do everything a jupyter notebook can do, you just need to write a bit of communication code to hook up a websocket. (For me sandboxing the ui is a feature not a bug.)
And you can also self host it on a machine of your choice, and edit it locally.
We do that internally for all our dashboards.
At Splitgraph [1] we're building a "data delivery network" (DDN, like a CDN but for databases) that looks like a big Postgres database. It works really well with the Observable Postgres client in private notebooks -- you configure it like you would for any Postgres connection. For public notebooks, you can use our HTTP API for sending SQL queries directly to the DDN. Here's an example [2] using that SQL-over-HTTP API to plot some Covid data.
[0] https://observablehq.com/@observablehq/connecting-to-databas...
[1] https://www.splitgraph.com
[2] https://observablehq.com/@mildbyte/splitgraph-ddn-oxcovid19-...
The idea is not surprising: using database indexes and a backend to send the frontend necessary data to render on demand. DB and the backend are containerized so the installation, data loading and authoring are all one command.
For creators, we offer D3 and JSON-based declarative primitives that enable creation of pan/zoom like visualizations very easily (e.g. 10s of lines of JSON for a 15-zoom-level vis). You can check out an interactive demo here, which visualizes 1.88 million wildfires: http://wildfire.kyrixdemo.live/
Since big data vis is very use case dependent, I'd like to reference two other tools that might be useful:
Falcon: https://github.com/uwdata/falcon NanoCubes: https://nanocubes.net/
Disclaimer: I'm still in academia so I'm apparently biased towards academic works. But fortunately they are all open source.
I checked out the Falcon documentation on Github and currently don't have a great understanding of (a) what it would be like to "write in Falcon" and (b) what it's intended use case is and how it differs from existing libraries.
Do you mind clarifying?
3 questions:
1. Why isn't there a demo I can just click and visit?
2. Why not TypeScript? (I think this would be a worthy short term and long term investment —for this project and beyond— to make the upgrade)
3. Why Docker? (This one I'm probably wrong about and am just a curmudgeonly old gray beard). Every time I see Docker I say "nope". I get it's use for massive clusters, but for running on one machine I dislike it (at that point I'd prefer just to run an image on a cloud VM).
1. In the original comment there is a link to a demo created by Kyrix: http://wildfire.kyrixdemo.live/ Are you looking for a different type of demo?
2. Honest answer: we are very understaffed academics who also need to write papers and theses. We want to convert to TS, that is just one item on the wish list.
3. We want everyone to be able to spin up a Kyrix app on their laptop using 3 CLI commands (you can try, instructions are in the README). With a backend and a database comes the cost of complex installation. Docker helps make everyone's dev environment consistent so it's easier to troubleshoot.
2. I get it. I never had the "pleasure" of being an academic, but got the chance to be a software engineer working alongside grad students for a couple of years and it felt like I imagine what taking a tour of a sweatshop factory floor would be like. (a huge exaggeration, but I saw a lot of low pay and lots of time on pdfs and not enough on code)
3. Yeah, I get it, but maybe allowing everyone to run it on their own VM would be better (and providing a one click "click here to get your own droplet on digital ocean running kyrix" sort of thing). I've found that Docker suffers from the XKCD problem (10 different environments to support—what if there was just 1....11 different environments to support!). I've almost never had a pleasant experience using Docker (and anything that requires downloading a 1GB+ image I don't consider pleasant). But again, I'm probably just being a grump and the other grads I worked with all seems to like it. Another idea is could use do this with SQLite? Have a more slimmed down version that didn't require Docker + those dependencies
Thanks for the answers! Very cool stuff and it's a very interesting problem.
An interesting thing to me here is the layering of DSLs. E.g., SQL enables user-defined functions like filters, that push down to multi-GPU columnar analytics with the rest of the pipeline... and GPU arrow dataframes for zero copy / streaming to combine it all together. People are posting in this thread about 1M rows, but this stuff is built for 1B+. The DSLs means both analysts and devs work at high levels, and underneath, supercomputing.
Fun historical note wrt JS vs Python for GPU: both have different strengths/weaknesses.. but are basically fine-enough long-term, with tweaking. We started w/ proving out JS GPU dataframes on OpenCL to be more open + viz friendly, and after Nvidia liked using our platform, their RAPIDS.ai team spun up to bring the idea in a more corporate controlled & funded way to Python. That's where the community resources are, so we jumped on board, and every month is now quite a trip. GPU SQL, GPU streaming, etc :) JS does inlining and async better than Python, while Python has the data ecosystem, so I've been eagerly anticipating JS folks stepping up where we had to leave off.
It's exciting to see it all come together -- imo, still early days for what's possible!
Btw, if you referred to the 1M-row example I posted - we actually can do much larger than that. A recording of a visualization with 1B reddit comments is here: https://youtu.be/ccES97ni_vI Behind the scenes we do indexing with Citus, which is a distributed version of PostgreSQL. Our cloud budget can only afford hosting a small demo 24/7 so that's why. Also, because OP talked about loading data into the web vis tools - 1M can already break many web tools out there.
You can tell our target use case is different than yours, and than OmniSci's. It's great to see solutions being developed in a one-size-doesn't-fit-all world.
- omnisci SQL. in contrast, rapids.ai opens layers below (cudf arrow, dask, ...) that enable cooperating solns on top (blazingsql, cugraph, custreams, prefect, ...) that are faster + easier for their domains, w fallback to general dataframes/sql.
-- omnisci is governance by a VC co, while rapids.ai is by nvidia (who wants to sell hw, not sw) and more OSS partners
Omnisci did good engineering, so it does have strengths. ex: its geospatial visual analytics means it's a good esri alternative consideration, as it is more polished than manually stitching together cuspatial + blazingsql + leaflet etc. Likewise, commercially polished for hostile enterprise environments (procurement, ...).
re:scale, see rapids tpcx-bb numbers ('big data'), I think on 10TB datasets. it shows scale + cost effectiveness wins vs others. less obvious, out-of-core so can do TBs even on one GPU, and full tpcx-bb needed the above versatility where sql is a kludge.
re:graph vs table, if you do just points and no edges, the node table is just a regular table you can do regular tabular data analysis + viz in. ex: load in samples scored by some ml model (x/y plot w lots of data columns for each point), then connect nearest neighbors to make it into an interactive graph. we are doing more and more here in practice, it's fun :)
Overall you can see its evolution from prior sites- someone spent a huge amount of time demonstrating cool visualizations, then tried to make it all "proprietary". Nothing wrong with that, but why not make people addicted to the service by giving them the raw drug, rather than making them hate you for dancing through hoops?
Again- huge fan of access to many cool visualizations that have greatly improved what I can express visually for presentations etc., but very unhappy that I can't do this so much easier for my particular workflow. I'd love to be shown otherwise, but I've also spent a lot of time looking for answers, reading forum responses, etc., and they seem to all reflect wanting you to go through them, not along for the ride with them.
For instance, I host content on my personal website [0] that is written in using Observable, and in fact the site provides guides to allow anyone to export content[1] and embed elsewhere online.
My content can be entirely under my control if I save my libraries out of the Observable API as a tarball[2].
I genuinely think the differences between Observable and 'pure' javascript are driven by the vision for a more powerful way of using javascript for rapid prototyping, rather than any attempt to make it proprietary. I think these ideas are revolutionary and Observable is one of my favourite working environments as a data scientist.
I also think that Mike Bostock has shown good faith, and dedication to the open source community through decades of work, which he's always given away for free.
With all that said, if ObservableHQ and its API were to disappear overnight, the lack of an editing environment would make it difficult to update existing content and so I think there is space for a offline version of the editor. As far as I understand it, nothing's stopping a third party building this.
[0]https://www.robinlinacre.com/energy-usage/ [1]https://observablehq.com/@observablehq/downloading-and-embed... [2]https://github.com/RobinL/robinl.github.io/tree/dev/tgz
To clarify where I'm coming from, my use-case is (fully) offline. I want to mock the visualization in the service then get the raw, static JS offline to be used as a template. I'll write a Ruby wrapper around the data I'm exploring and gradually parameterize the template as I better understand it, and what my data looks like in it. I don't want to dance with details of the JS, I want to dance with details of the data and parameters, so get me out of requiring online hooks ASAP. I know this is possible because prior versions of Observable let you do exactly what I want, but now I get some super friendly/not friendly interfaces that obfuscates what I'm familiar with.
As a separate issue not fully related- Ultimately my goal is a static visualization for scientific publications or presentations (I can export SVG from Observable, so my prior arguments doesn't hold here). Science has a very long shelf-life, in my field we routinely reference papers and their figures 200+ years old. While I absolutely love the idea of "living publications" I also fear life that depends on other's services. If the big people can't play nice (coughing in your direction Google services gone dark, I can think of various examples that have screwed over scientists), then you can see my caution in adopting Observable's all-in-dynamic approach.
Confession though, I might have been powered by Mike Bostock fanboy juice. d3 is part of what made me fall in love with programming.
I'm building something similar called Starboard Notebook[0] that has a different set of trade-offs. It ends up being something in between Jupyter and Observable:
* One of the goals is to build Jupyter how it would have been if it was designed for the web (only).
* It's open source [1], plays nice with git (the format is plaintext), and supports local viewing & editing [2]
* Because of that you can host it yourself, put it on your blog / github pages, anywhere.
* There is little magic, it actually is just Javascript at it's base. This means you can use standard browser APIs and HTML, and when you are ready to "graduate" the notebook implementation that should be straightforward. In my eyes notebooks are only for the first 20% of the work that does 80% of the job for small applications/experimentation (which is often where it ends anyway).
* You can "build the ship as you sail": you can load new cell types dynamically at runtime. This is also how Python is supported (through WebAssembly).
* You can have interop with Python and Javascript which is really powerful. Example: Create a drag an drop form using HTML+JS, then process the dropped CSV file using Pandas and visualize using matplotlib.
[0]: https://starboard.gg [1]: https://github.com/gzuidhof/starboard-notebook [2]: https://github.com/gzuidhof/starboard-cli
micropip.install('torch')
Couldn't find a pure Python 3 wheel for 'torch'
Python works for stuff that was written for it purposefully. Some python libraries work great: numpy, matplotlib, pandas. But many others are not supported directly and can be installed through micropip, but that's quite confusing! These are the issues with Pyodide currently:
* All python code that is executed is synchronous, which means it can not make requests (or call sleep). You can actually make a request using pyodide.open_url('path'), but that makes a synchronous request which isn't really a good idea for anything but small files. I believe asynchronous Python is possible, recent versions of emscripten support it, but it needs someone to put the pieces together (which is not easy!)
* Some packages are huge without being split up. Scipy is actually the only one that's really problematic, I believe it's around 80MB? It should be possible to split it up (into scipy.interpolate, scipy.stats, etc)
* Micropip is asynchronous, so you get a promise when you use micropip.install('my-package'), but you can't "await" it.
* Loading Python initially freezes the browser for a second or two.. not a great user experience.
* Libraries which are not pure python currently need to be manually made compatible with patches - including torch.
Python in the browser (not just in Starboard) needs more love. This is powered by Pyodide[0] which has been making steady progress, but the project is without corporate backing since Mozilla's change of direction. Perhaps Observable can allocate some of their funding towards supporting Python in their notebooks too through this project? Also consider this a call to action for other contributors who want to see Python in the browser become a reality :)
Some more questions: 1) Assuming the bulk of folks use jupyter notebooks for analysis, why not integrate with jupyter or use d3.js in jupyter directly? I assume there would be a learning curve for data manipulation on js where analysts/scientists would prefer python 2) Assuming that this isn't even meant for the jupyter audience, who is it meant for within a company?
I totally agree with you that the DS community is much much larger than the vis community. With the new funding round I think they should start doing something with DS.
Given the marketing on the page, it also looks like it's targeted at business use cases (i.e. a bunch of data scientists who want to visualize something in d3 but also want to collaborate).
Their current target for paid customers seems to be infoviz folks at news outlets (the CTO and co-founder was at NYTimes' data viz team for a number of years): https://observablehq.com/teams
That doesn't seem like a huge market, but perhaps they're starting with this audience segment and will be branching into more DS-type features (i.e. supporting a Python kernel, trying to be a more feature-rich Jupyter).
Can you install libraries (from npm?)
Can you build and share custom components, not just notebooks?
Can you connect to a web server?
Interactive vis is not very well integrated with the frontend ecosystem IMHO. Integrating a very interactive D3 vis with React, for example, can be painful.