Comment by corytheboyd | Hacker News Reader

Comment by corytheboyd | Hacker News Reader

Parent Full thread

A near real-time peer-to-peer piano keyboard visualizer for remote music lessons. Peer 1 plays their keyboard, and the midi data is sent to Peer 2 where a keyboard animates and sound optionally plays.

Project name is Midishare.

Got the idea after starting piano lessons about 5 months ago. It’s all over zoom, which works surprisingly well for music lessons on its own, but it’s difficult rigging a camera to show the remote person what you’re doing on the keyboard, as well as getting the sound to come through (if you don’t have a nice audio interface). There is still the issue of communicating finger position, not quite sure how to solve that one yet, or if it really even warrants a technical solution (again, you also have zoom to just communicate verbally, works okay for fingering)

The keyboard I use to animate playback is a 3D model which communicates the flow of playback surprisingly well, it’s at least a pretty cool accomplishment on its own!

I’ll launch it with a Show HN one of these days, within a month or two is the goal!

There's an existing shared piano site that does _similar_ to as you've described, but only utilises a 2D model. It might be give you some ideas or some things to consider to include/leave out of yours: https://www.multiplayerpiano.com/

There’s an app on sidequest (for the oculus quest) that can track your piano and finger moves. You can input a midi file and then it sees you playing it and shows your virtual fingers. It might be possible to do hand tracking like that with an iphone. Have you tried concerting sound from a phone microphone to midi?

Hey thanks for taking the time to chime in (pun fully intended)!

> It might be possible to do hand tracking like that with an iphone

Yeah I’ve been floating ideas like that in my head as well, in fact another commenter just pointed out some promising tech. I’m still on the fence about the value add compared to cost of implementation and accuracy. One reason I like sticking to MIDI only is that it’s much much easier to keep it consistently good (assuming underlying connection logic is sound). The margin for error is so low here, you just can’t have the tech getting in the way of the music is what I am putting my bets on. I’m worried that hand tracking will end up being months of effort for something that ends up being about 75% accurate. Number is completely made up but hopefully my point comes across :)

> Have you tried concerting sound from a phone microphone to midi?

If you mean converting audio data to midi, I’ve thought about it and ran away quickly after haha. Whoever solves this problem to a high enough degree of accuracy to be useful will be sitting on something quite valuable. I have my goal posts set a bit lower at first, on things I am confident I can deliver to the public, but I plan to revisit this in the future :)

I get that this leaves non-midi keyboards behind though, as well as all other instruments. If anyone has ideas on that topic I am all ears!

Initially I do plan to keep the scope small to see how well it performs, identify the problem areas, etc.

For non midi keyboards complemented with a webcam, hand tracking could do the trick, maybe paired with basic audio analysis to get the timing right. Some off the shelf tools are available [1] and pretty decent. It does go against your plan to start small though, but if you're interested feel free to shoot me an email (in profile)

[1] https://google.github.io/mediapipe/solutions/hands.html

Hey! That looks interesting I’ll check it out and add it to the list of topics to revisit.

Summarized in another comment but I am a bit worried about hand tracking for a couple reasons. Also mentioned in reply to another comment, but will summarize here for convenience as well!

1. While I know there is promising tech in this area (and I am very interested in it!) the tolerance for error while playing/performing music is slim to none. Will a solution end up being 75% accurate? 90%? Both are two low IMO. Pure MIDI keeps things in a relatively more controllable realm

2. Camera rigging. There are just so many variables here for things to go wrong/right. Quality of the camera, position, lighting. It’s challenging just finding a good method of positioning a camera even if you have the right equipment to locate it.

All that said, I’ll still reinvestigate this zone of ideas in the future and your insight is much appreciated! First, need to prove that I can make the simple case work as well as I think I can :)

You're right to be wary, I re-read your use case and hand tracking alone won't cut it. However I still think that hand tracking _combined_ with audio input can be precise enough (above 99%).

Here is an example of raw audio to midi: https://magenta.tensorflow.org/onsets-frames

Regardless of the feasibility, it won't be as reliable as direct midi input, that's for sure! I think your goal of getting things working for a simpler case first is right, and wish you best luck :)

I encountered a keyboard with a very primitive, single player version of that feature and I was playing interesting tunes in a weekend having repeatedly made zero progress in traditional lessons.

It's so helpful to just get into the flow of playing, then the things that teachers tell you make sense.

Yeah I can relate so much to that.

Most lessons we cover concepts that don’t immediately click until I next sit down on my own and just focus on playing. After getting past the familiarity building, and into the flow state, all the things my teacher said start to make more sense haha. Some of the time at least, but that’s what weekly checkins are for :)

It’s a slow process learning to read music in my thirties, but it’s just as possible as learning anything else, don’t listen to people who say you can’t learn music as an adult :)

Do you have a plan for showing dynamics? Would be fascinating to get a live piano roll of a performance that used (say) a color scale for the loudness.

I plan on experimenting with methods of communicating velocity this weekend :)

The keyboard highlights engaged keys with color already, so I have that to play with as well as the speed and fully-depressed angle of the key itself.

VMPK is a big influence, which respects note velocity. https://vmpk.sourceforge.io/

I wonder if this would be a good application for Omniverse

I’m not familiar with omniverse, mind elaborating?

Reply on news.ycombinator.com