Let’s take a detour from our previous discussions and explore an exciting possibility.
I’m considering creating an open-source multichannel pitch-to-MIDI converter with 8 audio inputs, one MIDI output, one USB-C output. This device would convert audio signals from multiple sources, like divided pickups (including our Nu Multi, piezo pickups, and the Roland GK3), into MIDI (MPE and MIDI-2) in real time, with minimal latency. It will be a high-quality, customizable solution for electric guitars, basses, and other instruments.
I’ve developed the core pitch detection techniques over the years, including onset and offset detection, as described in the “Pitch Perfect” series (links below). However, optimizing for MIDI conversion still presents challenges, and community contributions will be essential. While I’m equipped and willing to tackle this, it’s a complex journey. As Andreas Szalay, creator of the Fishman Triple Play, noted, it took six years to release the product, and many years later after release, software issues persist. I can lay the groundwork for a multichannel audio-to-MIDI solution, but fine-tuning for various applications, instruments, and sensors will need broader community involvement.
I started a Github project. To succeed, this project needs community interest. This is a nascent project, and the initial feasibility assessment will be gauged on engagement, so your involvement is crucial in these early stages. If this idea resonates with you, give the project a star, and join in and introduce yourself. Share your background, your interest in pitch-to-MIDI conversion, and whether you see yourself as a potential user of this product. This will help us gauge community interest and understand how many people might benefit from this project. I’ll set a modest target of 500 interested people, and let’s grow it from there.
And by all means, spread the word!
I’ve been ambivalent about audio-to-MIDI conversion, opting instead for direct synthesis without the intervening conversion to MIDI. Traditional MIDI guitar synths do not capture the tonal character variations resulting from guitar playing techniques such as forced and natural harmonics, finger position, finger and pick attack. However, advancements in MIDI MPE and MIDI-2 have shifted my perspective. MIDI MPE allows for individual note control, making it possible to retain much of the nuance and articulation of guitar playing, while MIDI-2 provides higher data resolution and more dynamic expressiveness.
Hi! I’ve been following your work with great interest!
After mulling the maximal performance of MIDI for a while I have finally personally reached the conclusion it is best to completely forgo MIDI and focus on control voltage and digitally represented control voltage.
To this end, I have invented a data format- ‘timed samples’, which as a concept simply adds a second channel to audio data that contains the duration the value is to be held. With this, you can have midi-like stepped data and variable sample rate audio like data or actual audio all in the same data format efficiently, at the price of doubling data per channel but saving in sparseness. Here is a supercollider example.
Of course the price to pay is compatibility with existing controllers- but to my particular sensibilities, this is unavoidable. My Linnstrument is lovely- but does feel a bit mushy.
For hardware, to me, this perspective means representing sensor data as audio and exposing it to audio software that way.
Of course, this is slightly utopian and ignores the perspectives of marketability- there is of course a good case to be made to maintain the MIDI compromise. On the other hand- the potential of unifying audio and control data is considerable!
I hope this was interesting- thanks again for doing your work and sharing it!!
Carlo
Separating onset and offset detection from pitch tracking is a useful way to frame the latency problem, especially when each string has its own channel. For a non-realtime workflow, a clean phone recording can also be passed through an M4A-to-MIDI converter and corrected in the DAW, while your divided-pickup design is clearly the stronger choice for expressive live control.