Show HN: MRuby-Zest – A Scriptable Audio GUI Framework
log.fundamental-code.com
log.fundamental-code.com
Two years back or so I was trying to build a new user interface for the ZynAddSubFX synth and getting the level of customization I wanted while being able to embed the interface cleanly into plugin hosts was difficult. I could have gone the route of producing a series of raster based assets and providing fixed locations for everything, but since I had different devices with rather different resolutions I wanted something to be more flexible than that. After starting with Qt prototypes I borrowed the ideas of Qt's QML and applied them to a ruby based widget system (I would have gone with com mon-lisp + CLOS, but that's not the best route to entice contributors), a nanovg/OpenGL rendering system, and a open sound control based messaging layer between the GUI and backend. Initially there was a GLPK based layout system, though that was unfortunately scrapped due to the runtime cost.
The end result is MRuby-Zest and it's a system that's reasonably efficient, embeddable, and in my opinion fast to iterate upon designs. The last note is mostly due to the fact that the system can hotload code updating existing widgets in a live GUI. That hotloading combined with the fact that widget code is all ruby (rather than building off an existing C/C++ toolkit) has made working with the framework pretty fun for myself. Overall the project is still pretty young and somewhat tied to the Zyn GUI (named Zyn-Fusion), but I thought some of the design choices and trade-offs might be interesting to the broader HN community.
Outside of the technical choices this work is some of the first open source work that I've seriously tried to get community funding for. After working on the software for a few (~3) months fulltime I was able to put binaries up for sale and I've managed to get ~200 sales for my efforts.
Have you thought about bringing in Crystal when it gets more mature?
Congratulations on the sale and good luck with the project.
Did you reimplement property binding ? That's the #1 feature of QML imho and I couldn't find an example in your source.
As mentioned at https://github.com/mruby-zest/mruby-qml-spawn/blob/master/mr... the properties form a dependency graph, so the property bindings should stay up-to-date when you change dependent properties.
This is one of the features of the overall framework which may be a bit under-tested at the moment. I've fixed a number of bugs in the system, though you certainly can get into odd edge cases when updating bindings during widget creation or destruction.
hmmm... but what I don't see is stuff like `onColorChanged: doStuff(color)`. Basically, the event system ?
Elsewhere in the codebase you'll see whenEvent style callbacks https://github.com/mruby-zest/mruby-zest/blob/master/qml/Val...
Most of the events, however, are handled through more traditional methods as that ended up being more comfortable when scaling to a larger program.
- Embed-ability: Qt can be used to build a plugin interface, but when a plugin host (which may be implemented in some version of Qt) loads multiple different plugins (which can be different versions), then it's possible to see conflicts with the global data that Qt creates. These problems are fairly hard for typical users to report and debug. IIRC this can be worked around with static linkage, though that opens up a separate can of worms with respect to Qt.
- Licensing: For the initial fundraiser the plan was to release binaries of the GUI for a fee with the full source being released at a later date. Doing this was simplified by using custom tools which were almost all under the MIT license.
- Layout: QML tries to specify widget properties in terms of fixed sizes of the leaf nodes and then using bottom-up algorithms to place the parent widgets. The initial system in zest was to use a constraint layout framework to specify all system constraints without hard coding pixel sizes for any widget (other than the top level window as specified by the window manager). Incorporating this different approach did not mesh cleanly with QML's existing abstraction
- Remote Data: QML works great if you're going to hook onto an app driven by Qt classes and the associated signal-slot system. Unfortunately Qt's approach does not work in a 'realtime-safe' manner. That is to say that if it was deeply integrated it could cause faults where the audio processing would be delayed generating audio glitches due to the timing being incorrect. When working with the OSC interface that existed on the target app (ZynAddSubFX), specifying parameter bindings felt rather unnatural and once again, didn't mesh with QML's expectation of how events should flow.
- Javascript: Personally I'm not a fan of javascript, so developing in a different scripting language made the process faster in Ruby.
- QML's C++ API: At the time I started work prototyping the GUI I needed to occasionally work with QML classes from a C++ perspective and sometimes extend/create new classes there. The C++ API was under-documented and the API felt clumsy. This is something that the Qt documentors have likely resolved at this point, but at the time it was a limitation.
- QML's mental model: I have worked with Qt in the past, though not all that extensively. Qt and QML work very well if you stay within their own realm, but once you start going against the grain, you need to understand the internals fairly well. I found it difficult to learn about the internals and as such for a long development process which needed to have fast iterations, it seemed like a better fit to work with code that I was more familiar with even at the cost of needing to write it myself. In the end I think this was mostly justified by the relative small size of the 'framework' vs the size of the widget code within the overall Zyn-Fusion GUI written using mruby-zest.
I know there were a few other issues at the time that I decided it would be more productive to build a framework along the way, though those are the items that come to mind first.
Ah, cool. I remember siting in on part of the 2017 LAC workshop for i-score (i-score was renamed to ossia score, right?). Once you remove the embedding criteria Qt itself becomes a much more feasible option.
I haven't used score, much, though I should likely play around with it given the amount of OSC work that I've done around zyn.
By any chance, have you looked into OSCquery (https://github.com/mrRay/OSCQueryProposal) ? It's a protocol we're currently a bunch working on and which is starting to make some inroads even in paid media software ; it greatly facilitates interoperation between multiple OSC-compliant software since it allows multiple software to easily query all the OSC nodes of each other through simple JSON over websocket communication.
One of the limitations for ZynAddSubFX is making discovery a multi-step process, since there is a very large number of OSC paths (if you count out all enumerations it's ~4,000,000). So flat listings don't really work and even if that's resolved the rest of the information needs to be presented in a compact/efficient format for the system to be performant. I'd say the OSCQueryProposal looks like it falls back on HTTP concepts and javascript conventions too heavily and adds more verbosity than I'd personally prefer. Looking over the options that I had considered in the past I'm looking at libmapper again for some possible future ideas and work.
Aside: Did you know the FUDI protocol(0) used in the pure data language is dead simple, if you wanted a faster bridge than OSC into/from that language.
I'm biased towards OSC however since the binary serialization is very easy to manipulate in a realtime context (i.e. no locks, no dynamic memory). In fact I was biased enough that I ended up giving a talk at the linux audio conference about realtime safe OSC http://lac.linuxaudio.org/2018/pages/event/39/ (with some nice performance figures to boot). It's cool to know what puredata is actually using under the hood.
I should clarify that when I said FUDI is faster, I was repeating the grumblings I've heard on pd's email list, despite not having benchmarked anything myself. With tcl's "everything is a string" (0) format, I could imagine this being contextual. But I am really curious.
(0) - https://wiki.tcl.tk/3018
You might find this kinda funny, a piece of pd's documentation regarding the osc objects:
"OSC is a complicated networking protocol (FUDI, as used in netsend/netreceive is simpler and better but less widely used). oscparse and oscformat make no attempt to deal with timetags or aggregates of packets, nor with streaming OSC. Also, no attempt is made here to clearly distinguish between the OSC address (symbols) and the following data, nor between blobs and lists of numbers - it is assumed that you know what types the message should contain. You can alternatively use the OSC objects from mrpeach which have more features than these."
https://media.ccc.de/v/lac2018-39-rtosc_realtime_safe_open_s...
Right now if you get the GUI running by building from the mruby-zest-build repo, then you'll be able to start the interface and it will load up MainWindow.qml as well as child widgets.