Sunday, November 11, 2007

touchscreens and the approaching tipping point for musical instrumentation

Touchscreens have been around for quite awhile, but they're starting to get seriously good, offering decent response times from devices scarcely bulkier or heavier than an ordinary LCD screen. Pressure sensitive screens are just around the corner.

So what do touchscreens have to do with music? They make instantly reconfigurable, virtual control panels possible. I'll revisit that point and connect the dots later.

Take the iPhone for example. The iPhone was far from being the first device to use a touchscreen, but it may be the first to make really good use of one, combining vivid graphics with multitouch technology and gesture detection. While it uses a menu system much like phones have had for years, navigation through those menus is practically effortless, because current options are presented simultaneously, and choosing one is as simple as putting your finger on it. Each menu uses almost the entire screen to handle a limited set of options, maintaining a moderate density of information and presenting a moderate selection of controls. Each of those controls has a very specific purpose and action, unlike the overloaded buttons on an ordinary cell phone, which change what they do according to context.

So a single touchscreen can offer many different control panels, switching between them as needed, and the controls on those panels - buttons, dials, sliders, steppers, etc. - can have both recognizable identity and clearly visible state.

Now let's take a minute to think about musical instruments as control systems. Physical instruments have a static set of controls, which can typically be applied in various combinations - multiple notes played together on a piano, multiple valves closed/opened on a clarinet, multiple, independently fretted strings plucked or strummed singly or together on a guitar - but even those combinations constitute a fairly limited set of options. Woodwinds produce harmonics of a single fundamental; change instruments to change fundamentals. Brass instruments allow you to change the fundamental by changing the length of the resonant cavity, using slides or valved extensions, but their voices also change, and doing that on the fly is a somewhat awkward way to generate more options in any case. Guitar frets are typically arranged to provide a twelve-tone, equal tempered scale. They could have more frets, or could be strung and tuned to offer six steps between each pair of successive twelve-tone notes, but the latter would mean sacrificing range and all but the simplest chording, and the frets already get crowded as you reach for higher notes.

Electronic keyboards are considerably more flexible, with programmable processors that are more than adequate to generate any sound you might want an instrument to produce, but the options they make available are still mapped to twelve keys per octave, and any use which doesn't at least approximate that arrangement is unlikely to be intuitive. Other instrument-driven synthesizers are similarly constrained by the control device.

That's where touchscreens come in. A touchscreen can show you just the options you need for a particular purpose, arranged in a manner that makes good sense or is at least easily recognizable. It could, for instance, present you with just enough controls to produce the notes that constitute a particular scale. That much isn't new. What is new is that a system using a touchscreen interface could instantly transpose that scale, up or down, or just as quickly switch to a different one, with visual cues so your brain is able to follow the change. It could also offer two or three scales side-by-side, with cues to show which notes are common among them, and allow you to rearrange them by simply dragging them around with your fingertips, elaborate them, perhaps making a copy first, or make them shrink and run to a corner to keep them close and ready for use. When playing, note patterns could also be visual pattens, meaningful bursts of light, and both scale manipulations and played notes could be incorporated into loops and scores, which could also be displayed. Creating, altering, and combining such loops during performance should make for some interesting high-wire acts.

As a scale-generator, such a system could also be used to unify the sound of other, more conventional instruments, bringing them into tune with each other and with scales they weren't designed to produce.

There's some advantage in all of this, even if the only notes you're interested in using are those found on the piano or guitar, but if you're fond of combining pure intervals, you'll quickly discover that the variations are endless, far beyond the capabilities of any static interface. Touchscreens will make this vast set of possibilities the musician's playground, really for the first time.


  1. I have a MIDI controller, a CME UF8, to be exact, and this MIDI controller does not produce a sound.

    It is arranged like an 88 key weighted keyboard and has two slide controls on the left. One, the Pitch-bend Wheel, and two, the modulation wheel. These two controls are designed to send MIDI signals to my computer.

    What my computer does with those signals are determined by the software programmers or the scripts I apply to my virtual instruments.

    In addition, the keys measure a MIDI control called velocity and a separate MIDI control called aftertouch which is employed by depressing the key further after pressing the key initially.

    I will agree that a touch screen as a dynamic controller is very robust--but it's not the touch screen that determins the robustness, its the design itself--as evidenced by all you can do with a mouse.

    Of course, the iPhone is a handy example, both literally and metaphorically, because its controls ARE its touch screen--but I don't see how this will necessarily improve the potential of music.

    A touch screen is simply a controller matrix: Width x Height x Presasure.

    My keyboard is modeled after a Piano but half the time is not meant to control a piano sound, and the pitch of virtual instruments I play are not restricted to the western 12-tone scale.

    The software design makes it dynamic, my controller simply controls that dynamic.

  2. Setups like yours work fine, if they work for you. If you can repurpose a piano-like keyboard and keep track of what the effect of pressing a key will be, during a performance, particularly if you switch between several such repurposings, more power to you.

    The reason I believe the touchscreen will prove revolutionary in music is that it allows the substitution of visual metaphors which are better aligned with the musical purpose of the moment. It is potentially any instrument you can imagine, using any scale, complete with visual cues.

    Frankly, I think touchscreens will find more acceptance sooner as controllers, rather than as instruments, but that's partly because there hasn't yet been much imagination invested in creating those instruments.

    I hope to contribute to changing that.

  3. I was thinking of just this idea the other day as probably as the most viable way for extended microtonality to take off. I envison being able to go from on generalized keyboard to another with the push of a button or other side keyboard by common tones or not where the keyboard could be repositioned under the hand automatically.