chenglong-hn

We find the issue is that the language agents use to create interactive visualization is kinda low-level. The logic for specifying data encoding, style and interactions are all entangled (if you swap x,y axis, you are likely going to redo your interaction and layout again). Thus, despite agents have been working hard, it is essentially doing lots of stuff bottom up. We decide to design a language that can separate the visual logic and the interaction logic to make specification much cleaner, manageable, and friendly (to both human and AI). The magic here is that we are now disentangle the logic at the spec level, but make the compiler to do the heavy lifting (sorry, compiler!) to wire up logics to support easy composition and adaptation of the logics to create a expressive interactive charts. We used a trick of bidirectional transformation to help the compiler maintain the mapping between pixel/visual space to data semantics so the specs can stay at a higher-level. We made a video to explain the ideas.

https://youtu.be/JAJy7ToczMs

Some principles we had in the design:

- the language should be high-level and the agents can easily user them in chats/apps declaratively (with many presets).

- the language should not give up expressiveness. We also provide a programming API, so that you can use it to create charts in your app/dashboards for external interactions. The expressive API also makes it possible to compose very cool bespoke interactions (the spinning globe in the website).

To try it out:

- You can try Flint in your agent app. Simply call your agent to create it as an MCP app in your chat! (https://microsoft.github.io/flint-chart/#/mcp)

- If you are building applications, ask the agent to use the library. Flint is especially suitable for building agentic apps where you want agents to bring up an interactive charts quickly in the user session following their asks.

- Btw, the flint mcp app allows you create interactive dashboard in the chat as well.

Finally, I'm thinking that despite AI agents are now so good at coding, that they seem to be able to create EVERYTHING. It might not just be the time to give up designing new languages to make it easy to describe and create stuff. In the end, language is still the way for us to communicate with the computer, I think the alignment of programming language between human and AI in could probably make building, steering easier.

Would especially be handy when we are building some agents that produce charts that serve end users (they want faster and more reliable experiences!).

But! We have a new project coming in to work on annotation machanism and interactivity to make it supporting presentation tasks.

I added an issue to track this

https://github.com/microsoft/flint-chart/issues/48

https://github.com/microsoft/flint-chart

Effectively Flint is designed to specify charts using another set of semantic parameters over geometric parameters, which happens to be something AI are pretty good at! (They can based on field names + sample values to infer the semantic types reliably than guessing geometrical params.

Btw, for others reading this comment, this is the XKCD we are talking about! https://xkcd.com/927/

Good news is that AI do make new language a bit more accessible than before! If your agents can use it well and you can steer easily, it will naturally be good adoption.

This is exactly why this is an intermediate language designed to get 95% stuff right easily (for expressiveness and reliability purpose), while 5% of more advanced case where the agents need to revise chart for other purpose can be done easily on top of the compiled low-level spec (low in terms of Vega-Lite etc, not SVG). We are not really designing a higher abstraction to replace existing ones.

In the past, the split is like 50% good at first run for some common stuff, all other stuff requires agent-loop or user involvement.

Our goal is to make it easy for most case, not everything needs a full multi-round trip agentic workflow to solve. :)

We are kinda all advanced users in fact, for a lot of users, they are easily get confused with the first time result if that is not as good, and the interactivity cost / multi-round isn't an option.

We use a fun elastic algorithm to decide dimension etc within the developer's constraint.

The "how it works" section explains a little bit of this. For example, for the heatmap example showing temporal data, for a "good-looking" chart, we need to (1) reconcile the conflict between banded discrete steps and continuous temporal axis, and it requires understanding and setting stepsize and time parser, (2) for the correlation color, we need to set domain etc under the color axis.

These are supposed to be handled automatically as system defaults, but the tricky part is that these decisions are "semantical", thus requires us to understand the data and design principles, thus existing languages won't stretch that far. And the actual good looking spec is actually over 40 lines of json spec with many low-level paramters, way beyond the simple 5 line encoding promised by GoG. Flint uses semantic type and a layout optimization algorithm to handle this, so 5 lines of encoding + data semantic types can derive rest parameters automatically.

Some examples in the gallery are more extreme: like the waterfall chart example is way over 100 lines of code, and sunbusrt, rose chart are even more since compositions are quite difficult in GoG.

Glad to have a discussion on this level! In fact, we wrote a paper about this, will be putting it online in a week or so!

A challenge with GoG is that it assumes configurations as second-class stuff, which makes it quite difficult for users to deal with things like changing formatter, scale, annotations. Flint kinda want to hide this aways (so Flint sets them on behalf of the agent or the user). But yeah, GoG is still the foundation for expressivenss.