Virtualsity: Inventing the instruments of the future | Hacker News Reader

Virtualsity: Inventing the instruments of the future

harpers.org

18 pointsprostoalex12 comments

If you're taken with the Opals hexagonal note grid but aren't picky about it being made out of Walnut burl, you can get an Axis controller built on the same principla for a fraction of the price: http://www.c-thru-music.com/cgi/?page=home

It's interesting that a lot of these instruments are electronic recreations of pre-existing instruments. As a saxophone player who has played an EWI, the electronic versions take time to learn and often end up feeling not as expressive.

I think the title would have been better if it had said musical instruments. My first thought was that this was about measuring instruments of some kind. Oh well.

Making new instruments is pretty hard stuff. One thing that I've noticed lots of new instruments seem to flake on is making the instruments really, truly expressive. At a simple level, most new instrument ideas I've seen seem to basically be a trigger sound on/off and if you're lucky some kind of volume control and if you're really lucky a pitch bend -- all tied into some kind of synth.

But most real instruments offer a tremendous amount of rich flexibility in performance. I grew up playing the violin and one of the reasons you don't really hear good synth approximations of a string instrument is because on a real instrument there are thousands of subtly different ways of playing a given note. Subtle differences in attack, bowing, bow position, string position, pressure of your fingers on the strings, incredibly slight intonation changes, and on and on and on. I can say with some confidence that no two notes played on a violin will ever be exactly the same.

The example of the Continuum instrument is probably the most advanced I've ever seen, and even then there's all kinds of flaws in the sound, bits that sound noticeably synthy -- and I can think of a few performance issues that this instrument can't easily replicate. That's not to say some other kind of sound wouldn't be amazing (I was more impressed with the out and out synth sound than the string sound). They've tackled an impressive number of challenges.

This makes learning these kinds of instruments very hard, but it also means that they can be performed at incredibly advanced virtuoso levels.

This article does a good job at highlighting some of the major efforts to produce expressive instruments. It's pretty exciting to me that technology has started to reach a point that really expressive instruments, capable of virtuoso level live performances have started to show up. My personal favorite is the Eigenharp, but it also has a little bit too much of a one-man-band aspect to it what with the mouthpiece and all.

One point that the article makes however is that these instruments are still remarkably rare to see. I think one of the problems is that they are hideously expensive for the most part. There's no entry level cheapo middle-school music student version they can bang around on the bus like when you're learning an acoustic instrument. I think my first violin cost $150, and I got $100 of that back when I traded it in for a better instrument.

Musicians, being notoriously poor, often simply can't afford one of these things and thus can't build mastery on them from a very young age.

Some examples:

Continuum - $3400 - $5300

Eigenharp - $4000 - $8260 (pico version is $800)

Seaboard - $2000 - $9000

Linnstrument - not sure, but looks like between $1000-2000

Still, as a musician, I find all of these far more interesting than something like Imogen Heap's MIT medialab-like (of which I disliked everything I ever saw coming form there) wearable instrument tech.

Part of this is the complexity of the interface (I.e. Piano keyboard vs. violin, etc.) need to be matched by complexity in controllable modulation a in the synthesis engine. The history of musical synthesis was a piling on of complexity up to about the late 90s then there was stagnation (or to a degree even reversion.). Part of this was simply the ability for users to reasonably build sounds with some of the more complex synth engines, but I think a lot of instruments (at the time they were mostly sample playback subtractive synthesis) couldn't build the complex nuance of sound from a natural system. In the mid 90s Yamaha came out with a synth called the VL1. It was hugely expensive and used physical modeling waveguide synthesis. It modeled a vibrating column of air in a wind instrument, and to program it you needed special software to design the construction and shape of the instrument (this was radical because all synthesis at the time was about controlling the sound.) It was actually possible to create instruments that couldn't play in tune, or wouldn't even make a sound. This instrument really responded well to complex inputs and was especially sought after by EWI wind players (the original was a keyboard instrument, but required a breath controller to play most sounds since the sounds were designed to require constant energy going into the system to make sound.). But it turned out to be exceptionally difficult to make musically useful sounds, and unlike most pro synths of the time, you couldn't program the models until some years later yamaha released their internal sound design software.

Since then most commercial synths have either been multipurpose sample subtractive, or virtual analog that simulate the old analog synths from the 70s.

Yeah, looking at the Continuum demo, it looks like thousands of hours went into trying to build up a virtual string instrument sound. Somebody has to splice the samples and build up all the velocity levels and all that.

I'm definitely not disparaging it as a ton of work. But I think it also gives us some ideas about the limitations of our models of how sound is actually made. Along the way we've ended up producing some truly unique and interesting sound models which have given birth to any number of entire musical genres.

I think part of the problem is also build some kind of control interface which can even detect the elements of a nuanced performance. Take something as straightforward as an electronic drum head. You've got velocity, edge to middle playback area, different parts of the stick of hand, the side of the drum (not just the head). Different stick head materials. You can stop a ringing drum or cymbal, or disrupt the vibration in subtle ways. For something that just seems like it's "onImpactPlaybackSample()" it's not anywhere near that simple.

I think that why the most successful electronic controllers have essentially been patterned after acoustic controllers. It's easier to start with a basic model of a saxophone, and then add wind pressure, embouchure, and other real techniques. That way you know people will take to them because all the wind people already know these techniques. History has a lot of unique musical interfaces that never got past curiosity because most performers couldn't adapt to it (theremin...)

I'm an instrument designer (Ambient Addition, One-Bit Groovebox, Exertion Instruments, PSPKick, etc.) and you're right. Jaron Lanier in the OP is right.

Traditional instruments evolved around expressivity of physical phenomena; particular configurations of vibrations.

Electronics (and computers) are great for information, but information doesn't resonate at audible frequencies, and it doesn't respond to human movement. Yes, theremin - but it's two-dimensional and it's not a physical system of the same species. it's an isolated physical oscillation with severely constrained external influence.

Synthesizers sound great, but only through heaps of patch design and only modulars begin to bring tangible structure that any simple string instrument has. And even then, players aren't manipulating the electricity in its own medium, but operating at an abstraction. Modern music has now nearly completely re-inherited the legacy of personal computing and thus we have one-man workstation music.

You're doubly and triply right about that awful trope of the one-man band and more importantly about the price scales of instruments, beginning cheap for kids who have more time and imagination than money.

My current approach is to regress to guitar-like instruments outfitted with embedded synthesizers. I recently showed one at a gallery show embedded in a cigar box. It doesn't have exhaustive sensors (only pressure and pitch), but it has other affordances which yank electronic music back into the domain of open-ended interpersonal vibration as opposed to vending machines for dance music .aiff files :)

Expressivity is often overly-focused on multi-modal, high resolution sensor inputs when it's really more about the ability and freedom to respond to the energy of a performance. Glove controllers usually fall down for the same reasons: they merely map a few parameters of an intricate process to cartesian movement like slot-car racetracks with no means to fail or they suffer from a lack of feedback - a symptom they share in all computer user interface applications - or they're simply too general and users (vs. musicans, ha!) spend all their configuring and no time rockin'.

Individual virtuosos aren't as important to me as seeing small groups of electronic musicians in true collaborative performance. My best example so far was in fact made with Imogen Heap's collaborator Kelly Snook: https://www.youtube.com/watch?v=BNgyMfH3E9Q A less precise, but larger group is here: https://www.youtube.com/watch?v=PDYabnu10qk

Great videos. Definitely not virtuostic, but fun. It's harder to define the kind of virtuoso performance in a group, which may be why full orchestra digital synths have made more convincing progress than individual instrument performances.

There are also some "easier" traditional instruments to target. A Hurdy Gurdy for example has far fewer modes of expression than a violin does and might be a nice physical model to base an instrument off of, in fact it's not too different than the Electric Eel you pointed out in your videos.

https://www.youtube.com/watch?v=K8zGZJQ5yps

Which of course provides an entirely different set of expressible performance options than even something like a Nyckelharpa

https://www.youtube.com/watch?v=7sfBcWvVUbs

> Yes, theremin

I've always hated the way theremins sound. There's an aspect to the sound, a buzzing sliding sound inherent in the instrument that I could never quite cotton onto. It reminds me a bit of electric violins which I toyed around with a little as a teenager. Something about the way they work eliminates quite a bit of the performance.

> the way theremins sound.

If I were to hazard a guess, I would say it may partly be due to the lack of transients, which again, goes back to the replacement of a physical process with an oversimplified informational model. In general, the ear is much more sensitive to the transient portion of sound than to the main oscillation. Many sounds can be synthesized with the same exact continuous waveform, but with different attacks, mainly due to the masking effects of the ear. Not to say Theremin doesn't have its place, but it has numerous issues and definitely deserves to evolve.

Thanks for the nyckelharpa video. I had definitely seen it before, but the significance hadn't been as strong then :) Seeing that inspiring performance in the context of this conversation was valuable!

In the past, I've been very conservative about avoiding corny and campy music tropes - there are way too many "wacky" novelty instruments, so I've avoided hurdy-gurdies, believing they had some kind of non-serious association (just listen to the name?!). As you pointed out, the leap from Electric Eel to Hurdy-Gurdy is almost trivial. So perhaps at some point I will loosen up and build some "digital hurdy-gurdies." In fact, one of my favorite instruments both conceptually and acoustically, is the autoharp. Perhaps I could combine the autoharp, nyckelharpa and hurdy-gurdy into something fresh.

In the meantime, I've been concentrating on that "transients and physical systems" problem. As we know, the traditional electronic envelope model, ADSR, is tremendously conservative and unashamedly uni-dimensional. Inspired by Venkatesh Rao's book, "Tempo," I'm trying to make a model of envelopes which is more like a weather system than bouncing ball.

> may partly be due to the lack of transients

It's possible. There's all kinds of issue with the theremin on the attack of the notes that drives me a little batty.

I used to play around with digital tracking quite a bit, and one technique I developed that improved the sound considerably was to start note playback at some random offset near the beginning of the sample. It made the attacks sound more complicated and "human" even without playing around with the timing of any of the notes. If I had a second sample from somewhere else on the same instrument, I'd also sometimes substitute it on the same note as the first sample, just to provide some variety in the tone during the sustain.

One other idea, bouncing off of your group performance focus, is to focus on folk or ancient music ensemble type music. The nature of the virtuoso performances in those kinds of milieus is very different and maybe less technical than in more classical venues.

I remember taking an interest in Celtic fiddle music and attending some jam nights at some area pubs. It was pretty fascinating, everybody just sort of brought whatever instrument they played and if they could add to the song just sort of did what their instrument did. At any one time there might be a guy with a kazoo, some fiddle players, a couple flutists, autoharp, hurdy gurdy, guitar, saxaphone, etc. I'd imagine some electric instrument (so long as they didn't overpower the group) would blend in well enough. It was pretty intoxicating and the random collection of instruments almost always sounded fantastic.

The best example I can think of for how it sounded was something like La Bottine Souriante, a group which basically brings a complete random collection of instruments on stage and plays the hell out of them following elements of a traditional Quebec style of folk music (but at a high level) https://www.youtube.com/watch?v=KYza0wxs7Ic

A smaller ensemble might be Väsen, who feature a Nyckelharpa in their music. https://www.youtube.com/watch?v=78hGmYwnNHg

With ancient music, the techniques are less well developed when you play in the original style, even on something like a violin. The bowing is greatly simplified, you may not even be playing with vibrato. Complex hand positions are reduced, playing on open strings (no fingers) is okay. As a musical target it's appealing for electronic instruments and seems to tolerate the transition better than more modern music.

And it turns out it sometimes sounds amazing:

https://www.youtube.com/watch?v=zi5-WRgA5GI

In fact here's a fantastic example on an EVI

https://www.youtube.com/watch?v=Wd1gKm6QyDw

Sometime after Bach a lot of modern performance technique was invented, and eventually you end up with Paganini, who's music is unplayable on any electronic instrument I've ever seen. But he's doing crazy things like left hand pizzicatos in the middle of a bowing section and alternating between the wood and the hair of the bow while playing harmonics and all sorts of madness. But even just Paganini Caprice No. 24 pretty much exercises just about 95% of anything you can do on a violin. (3:50 in the video I've linked below will explode just about anybody's mind the first time they see it and realize what's going on).

https://www.youtube.com/watch?v=vPcnGrie__M&list=PLB084C0DA1...

I grew up playing the violin and one of the reasons you don't really hear good synth approximations of a string instrument is because on a real instrument there are thousands of subtly different ways of playing a given note.

That's similar to what moving from acoustic guitar to sitar ($100-300ish for a starter) felt like. I'd still like a hang though ($5000) or a clone thereof ($1500-2000).

Reply on news.ycombinator.com