Visual Node Graph with ImGui
gboisse.github.io
gboisse.github.io
This looks really great! I'm not sure I agree with the conclusions, nor the need for a different system than what Blueprints or Geometry Nodes offer. Personally, I think both of those things are great save for the major (and infinitely complex) feature of "clean this board up". The nature of the graphs make it easy to see when code repeats or is doing too much, just so long as the graph, itself, is legible. So if you could have some smart snapping, to force devs to keep it tight, or if you could employ some kind of machine learning for cleaning it up into sensible data flows, I think most of the biggest complaints I've heard from people who actually use these systems, would be addressed.
That aside, I'm always open to a new way! I, again, don't think what was demo'd is any cleaner than Blueprints (especially not fond of all of those connection points. Bottom and top connections don't look tidy), but that is definitely in the eye of the beholder, and this appears to be a well-delivered first attempt, but a first attempt nonetheless, so I'll assume that whatever visual issues I have with it, the underlying scheme works well. Because, honestly, it looks like it does! As presented, I think the system would definitely work great and it would certainly make a few things easier than how they're handled elsewhere. I'll admit, 'curry it to a specific data type and then operate on that' is how I code anyway, so I'm biased toward liking it. But I like that methodology because I think it works exceptionally well, especially for work like rendering. So great work on both the full app and this whole new way of scripting!
I'm not sure machine learning is appropriate for graph layout. What a good graph looks like is already well known: no overlapping nodes, minimum line crosses, not too much 'white space'. But doing this layout automatically is very hard (NP complete?).
But, honestly, that kind of thing is the only valuable future for AI that I see, as a creative. I don't need it to write my code for me, but I could sure as hell use it to handle all the tedious bullshit that we've cobbled together as far as tooling, CI, formatting and any other "had to do it this way to make machines understand it" half-measure we've had to take in the past century. The AIs that we see, now, which learn from a huge dataset and then stay mostly inert until the next big update are more toy-like in my figuring. They are standards to measure against and proofs of all kinds of very interesting and useful concepts. But the lasting/useful AIs will be the ones that train alongside your work and ONLY on your work (or, let's face it, the work done in your company that will be shared amongst all corporate employees so that all work can be done more efficiently) in order to provide you with what you would be trying to do anyway.
Do you have any references besides Graphviz? I haven't seen much that does anything interesting in terms of weighted or constraint-based layouts.
https://saturncloud.io/blog/what-is-a-graph-autolayout-algor...
https://github.com/microsoft/automatic-graph-layout
https://github.com/kevtiq/digl
https://www.mindfusion.eu/graph-layout.html
https://reactflow.dev/docs/examples/layout/auto-layout/
What do you think 'tensor-based learning' or 'training data' would do for graph layout?
> What do you think 'tensor-based learning' or 'training data' would do for graph layout?
I didn't and don't have a comment on that.
I didn't and don't have a comment on that.
That's what I replied to and the question I asked.
If you want to group nodes by some context, why not just do it? This doesn't seem complicated to me. I don't know why there would need to be 'tensor based learning' 'machine learning' or even some one else's algorithm.
which I think is what is really required for interactive graphs.
Why would this be required for 'interactive graphs' and what does that mean? If you don't know of any graph that does this and you think it's required, do you have some definition of 'interactive graphs' that has never been done?
Ergo, vis-a-vi have you ever actually tried making a graph that is rearranging itself while you're using it?
Also what is the "AI" doing that can't be done with a normal algorithm? Positioning nodes in a graph would just be determining their x and y positions. How much time have you spent working with node graphs? Arranging the nodes is easier than arranging your keyboard and mouse, it's trivial.
People did; you didn't.
> Ergo, vis-a-vi have you ever actually tried making a graph that is rearranging itself while you're using it?
No one said anything about "while you're using it". Linters don't fuck up your text as you type it. It's a utility like anything else. And, YES, I have built a working version of one. It doesn't use machine learning/tensors at all, and it's a fucking nightmare to develop. I can see exactly where those things would help me. It's too abstract, to you, to understand? That's a shame! But other people seem to get it, so if you are having trouble, maybe try approaching your ignorance with openness and curiosity instead of cynicism and trivialization ('uncross some lines').
If you want to know how much time I've spent working on node edge graphs, you wouldn't believe me. But I can at least point you to the start of the web-component version I'm porting: https://github.com/catapart/Magnit.NodeGraph It's probably two years out of date at this point, as I've been integrating some of the algorithms for the specific need I have for these graphs (interactive dialog), but even from that anemic demo you can see that I know how to build the damn things. Again, you're trivializing for...some reason? I don't know. You seem to think you know more about me on this, and that's fine, but you don't really ever demonstrate that so I can't see your commentary as anything other than intransigent ignorance. Which was obvious from your first response, which is why this is my first (and last) direct response to you.
I hope you feel better about whatever it is that is causing you to have this reaction to a pretty obvious benefit that you, for whatever reason, seem too stubborn to engage with.
Your comment: cleaning up your graph as you code it,
I have built a working version of one. It doesn't use machine learning/tensors at all
Your comment: we could employ tensor-based learning methodology, like ChatGPT uses
How would that work on a technical level?
You keep throwing around insults like:
those that don't seek answers with an open mind will often fulfill their own disillusion
if the person you're responding to can't intrinsically understand what would be good
It's too abstract, to you, to understand? That's a shame!
But you seem to be melting down instead of answering the most basic questions about what you're talking about. How and why would you use "tensor-based learning" to layout a graph?
Again, you're trivializing for...some reason?
Because graph layout features were invented essentially right after people started using node graphs. If a node to the right of another node is connected to an input to the left and the node to the left is connected to the input on the right, their wires will cross.
so I can't see your commentary as anything other than intransigent ignorance
I asked the most basic follow up question to what you said and you think you're being persecuted. Where does the expectation come from to make some sort of wild claim and have no one at least ask how it's supposed to work?
What is all this stuff about being "stubborn" and "closed minded" when you haven't explained anything? Just back up what you're saying somehow.
whatever it is that is causing you to have this reaction
You mean asking basic follow up questions?
- Video
- Images
- Rich Text
- Animations
And a way to compose these blocks together into bigger structures with shader effects. Preferably in a compiled language such as C++, Rust or Swift. Does something like that exist?
The underlying C++ part of this is kiiinda independent of the rest of the app, at least the graph part : https://github.com/ossia/score/tree/master/src/plugins/score... - I'm sure it wouldn't be too hard to extract this folder specifically. Shaders use the ISF specification (https://isf.video) and the backend is through Qt's RHI so things are then mapped to metal, vk, d3d or GL.
You might want to have a look.
Video, Images, Rich Text and Animations are typically what we strive for. Check out maybe https://nopefoundry.github.io/nope.gl/usr/howto/renders.html and the other howtos.
VVVV itself is based on .Net and you can extend its functionality by either writing nodes in C# or importing just about every existing .Net library just by referencing it or installing it as nuget. No need for wrappers[4]. For rendering vvvv uses the Stride [5] game engine which comes with a really neat shader system / language which is basically a superset of HLSL [6]. In vvvv those shaders are represented as nodes and you can open them in your favorite text editor directly from the vvvv evironment, edit the code, save and the changed result will instantly be loaded in vvvv[7]. If you have any questions join the vvvv matrix chat:
https://matrix.to/#/#vvvv:matrix.org
[2]https://github.com/TheFuseLab/VL.Fuse
[3]https://visualprogramming.net
[4]https://thegraybook.vvvv.org/reference/extending/overview.ht...
[6]https://doc.stride3d.net/latest/en/manual/graphics/effects-a...
[7]https://thegraybook.vvvv.org/reference/libraries/3d/shaders....
This would make graphs a lot cleaner AND could be made browsable by keyboard.
Could you explain, however, how could the grid make node graphs browsable by keyboard?
Kind of like excel, how you browse multiple tables in the same grid.
As remark on the spaghetti coding, most visual programming languages do offer modules, and the ability to add some structure similar to digital circuit design, if anything they make it easier to visualize bad code when proper coding isn't followed upon.
I enjoy graphs, you can think of the world wide web as a graph. When you click a link you get a different perspective of that vertice.
My dream is to combine nodal rendering to animated field of view layout.
In all the attempts I've seen to create an interactive 3D graph the problem was that moving around didn't change the perspective and paging in and out of the nodal elements.
Navigating a graph where the graphs aren't moved relevantly while you walk around is not useful it's like manual labour of walking to the store, the nodes need to change perspective, animate into different variations as you move around and page in and out different edges and vertices.
The algorithm you use to move/page nodes in and out and around has to be dependent on what you're doing.
Huh? I don't get this. The example right below seems a lot like coding with nodes to me. Perhaps they mean it's less procedural, more declarative? But that's the case with all of these visual programming interfaces. At least until you run into the limits of the declarative sub-language.
-If they are restricted to a suitable problem domain (e.g. image/audio/data processing).
-If have the ability to do some scripting in a node, to cover specialzied cases the designer didn't allow for.
(I'm working on a node based data wrangling tool)
I have a Django app for building topical maps for content marketing, and I'd like to be able to visualise the topic maps in terms of interconnected nodes.
but you'd have to write communication to your django app. I would do that in JS and then pass events to the WASM layer.
however there are react etc specific libraries as others have commented
I'd need it to work well with touch, for phone or pad users.
Here is a video of what it was like around 2009. https://www.youtube.com/watch?v=ZPYy8lry2do
I think the core problem is that - unlike text-based programming - the editor is intimitely tied to the rest of the authoring environment and runtime. Sometimes it is the entire authoring environment.
Text coding benefits from standardised tools that are decoupled from the applications they serve and interoperability. You want to write your Rust in nano? Go right ahead? Want to write some lua that does bulk edits to your COBOL library. Fab.
More importantly is that we learn the tools and keyboard shortcuts and those skills transfer to new languages and environments.
The main issues would be mitigated by:
a) some standardized UI conventions
b) all visual programming tools offering lossless round-tripping to textual representation (it doesn't have to be tuned for human writability - it just needs to be sane)
B also helps with the other big flaw with visual programming - the way it breaks version control workflows. I heard from an Unreal dev that even opening a blueprint and highlighting a different node will change the file. That must be fun when it comes time to merge...
I could write reams on this topic. I want visual programming to be better - I think it has unique strengths. But currently it has two niches:
1. People that can't bring themselves to learn text-based coding.
2. Tasks where discoverability and immediacy trump all the disadvantages.
The textual representation doesn't have to be the primary way of interacting with the program and it doesn't need to be optimal for everyday use - but it should be sane and visually comprehensible when the need arises.
Some people write, others draw. Each method has its own strength and problem spaces.
Encoding a complex issue sometimes is just shitty with a visual metaphor. Other times writing code is a really bad idea. If you want to express “flow” or “connections”, filter or stream systems like modular synths, that’s when the visual metaphor shines.
You use whatever metaphor works best at the moment. One cannot replace the other.
The “problem” is that people want to push all of “programming” into the visual metaphor. That’s like pushing all of programming into only functional, only declarative, only imperative or only OOP (or whatever), but worse. That’s not going to work, but I guess people can try.
It could be a failure of my imagination. I’ve designed similar things though, and these weird cases (especially temporal cases) become extremely hard to represent more universally. I suppose you could offer a “timeline” to indicate how things would behave on a time scale, but even that… I just don’t know.
It also seems key to have a node-based editor work with one language or a set which map together extremely well.
Part of it is what people expect. You get pushed and pulled when it comes to designing things for the web. Partially by constraints, partially by people's desires and expectations. It's better than ever these days, but my career has generally had the greatest challenges presented on the frontend of web projects.
I find backend development a bit "smoother". I won't say easier, but... I don't know, the frontend of the web is a jungle and a bit of an uphill battle sometimes.
There's definitively some hard limitation in dimension, though, so there might be the idea that, if one can only show problems of a certain limited complexity visually, why not do all the lousy (text-based, relational/reference-based way) but universal way that at least can be stored and edited everywhere.
So a central question might be: can we have a universal gain from, let's say, a three dimensional notation that still does not limit the problem space to these dimension.
I suspect it would quickly become too complex, but there's plenty of people smarter than me...
Eg (A LEFT JOIN B) = (A ∩ B) + A
Maybe I'm missing something though.
[0]: https://enso.org/
After all, we use lots of comments to give some structure to text-based coding.
There was a post about https://github.com/samuelmtimbo/unit recently, which at least uses some kind of hinted auto-layout (besides the more propriety fancy)
I really want to see more graphical coding for years, but node/graph-based and blockly seem to be the only approaches that got sone traction so far. So I like this thread and it seems at the right place.
I'd wish to see more use of 2D and now 3D space for structuring, annotation and overall overview and navigation. Text plus a directory tree view can't be everything, especially since we have so many structural views for coding, runtime, versioning that basically are connected in our mind, but also could be in a space with more dimensions. Not sure if we could find/agree on a common denominator, though. Text is one because it's basically the bottom of common ground. Not sure if that's not just the trivial solution (ok, I am).
Does not sound like the core problem to me. Computers used to be like that: everything was tied to the OS and compiler. That didn't stop banks from developing pretty complex systems.
IMO, the problem is that visual programming, despite its name, is visually a mess. You can't read it, you can't orient yourself in it. It's not structured, for starters. And piping an input through a sin function takes up 10 times the screen space of Math.sin(x), especially if you then have to keep that node's output from overlapping other parts. And you always must search back where the input to a node comes from, whereas in textual programs, you can put that information in the variable name. It's just not suitable to our way of thinking in chunks.
A graph editor is a large undertaking, having a few generic ones would avoid so much duplicate work. Having a generic graph editor would also allow an extension system to develop around it.
Each application would just need to implement the specific server protocol which does game rendering etc...
Imagine I was a consultant who offered to sit and watch over your shoulder and comment only on syntax issues. If you make a syntax mistake, I will help you fix it; "you need a closing paren here". I will also offer syntax tips constantly while you work; "looks like your going to create a list, first choose what you want the first item in the list to be". You should expect to be a little slower in exchange for these consultations I offer. Would you hire me?
These are essentially the tradeoffs offered by visual programming. Slower work for syntax level help. And visual languages often have some premium cost to them.
People doing significant programming don't need syntax level help.
What I learned was that when a problem domain needed a visual representation they had already created it (using pencil and paper, decades ago). Think of schematic diagrams, piping and instrumentation diagrams and such. In those cases we were very successful. In cases where no such diagrams exist, we always failed to create one that stuck with the users. Text is great for code because is is extremely compact and takes advantage of things like visual rhyme. Uml, for example, can never achieve the same results
Most kinds of software don’t need these things but if you’re writing code that generates graphics or audio signals then you do, because you get to manipulate your program and see/hear effects in real-time, and also seeing how different parts of your program connects together is more important than giving names to them (where there may not be a useful name you can give them anyway)
Theoretically you might be able achieve these things with only a textual representation and it could be best of both worlds
What does an algorithm look like?
Herman Hesse alluded to this in his novel _The Glass Bead Game_, but despite decades of discussion and work, no one has made a convincing pysical representation of that system.
I love the concept, and have made some moderately complex attempts, e.g.,:
https://www.blockscad3d.com/community/projects/1430644
https://github.com/WillAdams/gcodepreview
it always devolves to screen size being out-paced by problem complexity --- one gets something of an inkling of this at:
https://scriptsofanotherdimension.tumblr.com/
Alternately, one can just break a project down into modules, but then the top-level view becomes the wall of text representation (albeit w/ nice lines or captured into pretty boxes) which one is ostensibly trying to escape.
I'd love to see someone succeed in this, and I've been using:
https://github.com/derkork/openscad-graph-editor
quite a bit, and put a bit of money towards:
Imagine asking Stackoverflow about a coding problem that involves a visual node structure and code attached to some of them.
So yes, the round tripping is essential, but it also needs people to understand and be able to talk about the code in its textual form.
I came to the conclusion that realistically these tools will always be severely limited unless they focus on being good automated visualisation tools of syntax that is text first.
I'd like to be wrong about that, by the way.
Visual programming presents huge challenges.
> Text coding benefits from standardised tools
That's the biggest, IMO. Some degree of text processing is essentially universal to computing after the early 1950's. As you mention, that provides a huge amount of prior art in editors, source code control systems, various kinds of messaging, display tools... the list goes on and on. Textual languages can natively reuse all of this. Graphical languages need some sort of mapping to text in order to use tools that do not quite fit the way they work.
The other challenge is the way visual programming complicates the mechanics of the job. Laying source out in (~) one dimension is hard, laying it out in two dimensions is harder. Naming a function is hard, drawing a representative icon for a function is harder.
You also lose the notion of linear ordering when you move from a list of statements to a 2D graph of nodes. This is useful when you're modeling a data flow graph or a state machine, but harder when you want to express a branch or a loop.
I can't imagine a situation where given the choice (and assuming sufficient familiarity to be fluent) I would choose a node-based system over writing code. As I said earlier - the main advantage is discoverability (plus a certain robustness against making errors which arise from the constraints).
Discoverability is cancelled out by familiarity and if I'm going to use a tool seriously then I will become familiar with it.
Every other aspect of visual programming is a disadvantage and in many cases a crippling one. The sheer laboriousness of constructing expressions, the lack of sophisticated refactoring and the amount of busywork tidying up nodes so they aren't a jumble of spaghetti.
I think these are solvable but not until there's a way to avoid reinventing the wheel for every single system.
It helps to think of unix pipes when working with Node-RED, but when working with state machines to think of wiring on a circuit board. I wrote down my ideas in blog[3] - if anyone has more representation, please share!
[1]=https://noisecraft.app/ [2]=https://nodered.org [3]=https://blog.openmindmap.org/blog/pipes-wires-nodes
Kudos to Omar, the author of imgui: https://github.com/ocornut