Sunday, November 11, 2007

touchscreens and the approaching tipping point for musical instrumentation

Touchscreens have been around for quite awhile, but they're starting to get seriously good, offering decent response times from devices scarcely bulkier or heavier than an ordinary LCD screen. Pressure sensitive screens are just around the corner.

So what do touchscreens have to do with music? They make instantly reconfigurable, virtual control panels possible. I'll revisit that point and connect the dots later.

Take the iPhone for example. The iPhone was far from being the first device to use a touchscreen, but it may be the first to make really good use of one, combining vivid graphics with multitouch technology and gesture detection. While it uses a menu system much like phones have had for years, navigation through those menus is practically effortless, because current options are presented simultaneously, and choosing one is as simple as putting your finger on it. Each menu uses almost the entire screen to handle a limited set of options, maintaining a moderate density of information and presenting a moderate selection of controls. Each of those controls has a very specific purpose and action, unlike the overloaded buttons on an ordinary cell phone, which change what they do according to context.

So a single touchscreen can offer many different control panels, switching between them as needed, and the controls on those panels - buttons, dials, sliders, steppers, etc. - can have both recognizable identity and clearly visible state.

Now let's take a minute to think about musical instruments as control systems. Physical instruments have a static set of controls, which can typically be applied in various combinations - multiple notes played together on a piano, multiple valves closed/opened on a clarinet, multiple, independently fretted strings plucked or strummed singly or together on a guitar - but even those combinations constitute a fairly limited set of options. Woodwinds produce harmonics of a single fundamental; change instruments to change fundamentals. Brass instruments allow you to change the fundamental by changing the length of the resonant cavity, using slides or valved extensions, but their voices also change, and doing that on the fly is a somewhat awkward way to generate more options in any case. Guitar frets are typically arranged to provide a twelve-tone, equal tempered scale. They could have more frets, or could be strung and tuned to offer six steps between each pair of successive twelve-tone notes, but the latter would mean sacrificing range and all but the simplest chording, and the frets already get crowded as you reach for higher notes.

Electronic keyboards are considerably more flexible, with programmable processors that are more than adequate to generate any sound you might want an instrument to produce, but the options they make available are still mapped to twelve keys per octave, and any use which doesn't at least approximate that arrangement is unlikely to be intuitive. Other instrument-driven synthesizers are similarly constrained by the control device.

That's where touchscreens come in. A touchscreen can show you just the options you need for a particular purpose, arranged in a manner that makes good sense or is at least easily recognizable. It could, for instance, present you with just enough controls to produce the notes that constitute a particular scale. That much isn't new. What is new is that a system using a touchscreen interface could instantly transpose that scale, up or down, or just as quickly switch to a different one, with visual cues so your brain is able to follow the change. It could also offer two or three scales side-by-side, with cues to show which notes are common among them, and allow you to rearrange them by simply dragging them around with your fingertips, elaborate them, perhaps making a copy first, or make them shrink and run to a corner to keep them close and ready for use. When playing, note patterns could also be visual pattens, meaningful bursts of light, and both scale manipulations and played notes could be incorporated into loops and scores, which could also be displayed. Creating, altering, and combining such loops during performance should make for some interesting high-wire acts.

As a scale-generator, such a system could also be used to unify the sound of other, more conventional instruments, bringing them into tune with each other and with scales they weren't designed to produce.

There's some advantage in all of this, even if the only notes you're interested in using are those found on the piano or guitar, but if you're fond of combining pure intervals, you'll quickly discover that the variations are endless, far beyond the capabilities of any static interface. Touchscreens will make this vast set of possibilities the musician's playground, really for the first time.

Sunday, September 02, 2007

Tenori-on

Tenori-on is a new type of electronic musical instrument, developed by Toshio Iwai in collaboration with Yamaha. It's interface is a grid of 16 X 16 LED-illuminated button switches, in a frame that contains a few additional controls.

This device goes on sale in the U.K. soon. There will be a lauch event this Tuesday, September 4th.

Click here to go to Yamaha's Tenori-on product page.

Sunday, June 03, 2007

of primes, powers, ratios, and representations

So what does all this have to do with music?

As was known to the ancient Greeks, there is a tangible relationship between a tone and other tones the frequencies of which are small integer multiples of the first. (The Greeks thought of it in terms of dividing a string into lengths which are integer fractions (1/2, 1/3, 1/4, ...) of the length of the string.)

There are also tangible relationships between the lower members of a harmonic series, for example between the 4th and 5th harmonics. These relationships begin to break down above the 7th harmonic (some would say above the 11th), except as their numerical expression is reducible to a ratio composed of low powers of small primes, in which case even a tiny interval, such as that between the 55th (5 x 11) and 56th (7 x 2^3) harmonics, can be musically significant. (Because there are no common prime factors between 55 and 56, the interval 56:55 is already expressed in least terms and is irreducible.)

Such ratios can be represented by a short list of small integers which are the powers of the first five primes (2, 3, 5, 7, and 11), the values of which may either be positive, if the prime and its positive exponent belong in the numerator, or negative, if the prime and its positive exponent belong in the denominator. Using this approach, the number 2 (2/1) would be {1, 0, 0, 0, 0} and the fraction 1/3 would be {0, -1, 0, 0, 0}. This might seem overly complex, except that it is a comprehensive representation which can easily accomodate any interval that might be considered musically significant by virtue of possessing a degree of consonance.

Moreover, the set of values which might reasonably be used in such a list of prime powers is quite small. The power of 11 would never be anything other than -1, 0, or 1. For the sake of flexibility, one might wish for the range of the power of 7 to be a little wider, say from -2 to 2. For 5 we might wish a choice of powers from -3 to 3, and for the prime 3 possibly even a range from -5 to 5. The lowest prime, 2, is a special case; how many octaves would you like to be able to span? If you want to be able to accomodate fundamentals as low as 1 Hz and at the same time accomodate tones at the very upper limit of human hearing, and to be able to reference either from the point of view of the other, you'll need a minimum range of -14 to 14. Let's call it -16 to 16 for good measure.

That's 33 x 11 x 7 x 5 x 3 or 38115 possible combinations of values, a number smaller than the range of an unsigned 16-bit integer, not that I'd really suggest using 16-bit values and a translation table just to save space, not with memory and storage as abundant and cheap as they've become, but it wouldn't be unreasonable to use an array of five single-byte values (signed char in C).

Note that what's being described by such lists of prime powers are intervals in an abstract sense, the value of the ratio between two tones, which can be applied to any tone where the resulting tone isn't so far outside human hearing as to be irrelevant.

Most compositions won't come close to making use of the full range of the set of prime powers {2^(-16..16), 3^(-5..5), 5^(-3..3), 7^(-2..2), 11^(-1..1)}, but what they may do is to make use of several harmonic series, the fundamentals of which are related by similar ratios. In such cases, it's always possible to reduce the system to a single harmonic series with a lower fundamental, but doing so is very likely to be less meaningful than using several harmonic series and low-numbered harmonics.

As a matter of pragmatism, in the programmatic context it would be preferable to define a collection of harmonic series by ratios relating all of their fundamentals to a single base tone, rather than to generate a string of ratio-related fundamentals as each section of a composition gives way to the next. That progression can easily be represented by reference.

But a harmonic series, even a set of them, is not a scale. To produce a scale it's necessary to specify which members of those series one intends to include, and again powers of primes can be useful, in this case only positive powers. Integers greater than one are either prime themselves or a product of primes, and harmonic numbers are simply integers. You can use limits on prime powers to constrain how many times each prime may appear as a factor, and thereby constrain the set of harmonic series members in use. An example would be {3, 2, 1, 0, 0}, which would generate a scale composed of members 1, 2, 3, 4, 5, 6, 8, 9, 10, 12, 15, 18, 20, 24, 30, 36, 40, 45, 60, 72, 90, 120, 180, and 360. This approach may prove adequate in some circumstances and not in others. It would be wise to maintain at least the ability to tweak the results manually.

So, to recap, ratios described in terms of powers of the first few primes relate the fundamentals of harmonic series to a base tone, as well as relating the members of those series to each other, and lists of limits to prime powers contrain the set of series members in use. With these tools one can define scales.

Friday, December 29, 2006

a note about nomenclature

In the previous post I've used the word "pitch" to refer to the frequency of a musical note, to make it clear that I was not referring to the voice of the instrument, which is more commonly recognized as being the province of harmonics.

Henceforth, I may revert to using "tone", or use them interchangeably.

musing about the problem space

It's not as though scales composed of harmonics are really all that unfamiliar.

Woodwind instruments naturally produce pitches that are harmonics (integer multiples) of some fundamental frequency, determined by the size and shape of the instrument's resonant cavity. Brass instruments also produce harmonics, and those with either valves or slides are capable of altering the fundamental frequency by modifying the resonant cavity, making available pitches which fall between those which would otherwise be possible. Generally speaking, the notes either can produce are mapped onto the equal tempered scale, although it's quite a stretch in some cases.

Non-fretted strings can produce any pitch whatsoever within their ranges. Orchestras, dominated by woodwinds and strings, may gravitate towards scales that could be accurately described in terms of harmonics, although the scores they work from are based on the equal tempered scale. Those scores may include pitch bending cues to encourage this, even if the harmonic nature of the tonal target is obfuscated to the point of being conceptually absent.

Pianos and most modern fretted instruments, guitars included, are designed to support the equal tempered scale, but many electronic keyboards can be programmed for a wide variety of scales, mapping their keys to pitches that are a little higher or lower than they represent in the equal tempered scale.

The twin problems have always been building instruments offerring a wide enough selection of discrete pitches to support a wide variety of scales without tempering, and creating a notation that represents harmonic relationships in music compactly and in a manner that can be read quickly enough for performance.

The historical upshot of the first of these was that a musician with a repertoire spanning many scales would have to carry many instruments. The latter meant that no such notational system ever became commonplace, so it wasn't really possible to write down the music; it could only be preserved by passing along the knowledge of playing it.

Programmable devices (keyboards...) are solving the former, and opening the way to address the latter. We can easily record sound these days, of course, but that's not the same thing. Say you had available a performance instrument (or interface to a synthesizer) capable of a wide range of pitches in any given configuration and of being reconfigured on the fly, how could a composition best be represented to enable you to play while reading along.

While some variation on standard musical notation, read left to right with higher notes appearing higher on the staff, might be made to work, I think the same technology which is delinking sound production from physical constraints may also provide some interesting possibilities for the presentation of music as a performable abstraction. I have some ideas along these lines, but nothing yet so well developed that it doesn't seem better to leave it to your imagination.

Given a system which represents pitches as integer multiples of the frequency of some fundamental, or in all but the simplest cases, as integer ratio (e.g. 4/3 or 5/4) multiples of the frequency of some reference pitch, what sort of notation would encapsulate such relationships among the simultaneous/sequential notes that comprise a musical piece such that someone familiar with the notation could follow along quickly enough to reproduce the piece while reading?

Saturday, December 02, 2006

back burner != forgotten

Rumor has it that Apple has a tablet computer with a touch sensistive screen in the works. This is very interesting to me because it could simplify creating user interfaces in software for virtual instruments which present their tonal options in terms of harmonics, rather than force-fit to the equal-tempered scale. It might even be possible to create a usable performance instrument within the constraits of such a device, without having to migrate it to a larger display or custom-built gadget. I certainly intend to try.

Meanwhile, you can learn more about harmonic-based music at the wesite of The Just Intonation Network.

Thursday, September 28, 2006

on the back burner

While there's not recently anything (like a new version of the app) to show for it, there's not a day that goes by that I don't at least give some thought to this project. I'll eventually get back to writing code for it.

A contributing factor in my not doing so just now is that there's a new version of the language (Objective-C 2.0) coming along with the next major version of Xcode, which should ship at about the same time as the next major version of Mac OS X (Leopard, a.k.a. 10.5), due sometime during the first half of 2007.

While details of Objective-C 2.0 aren't yet public, I've seen enough to make me think I'm going to want to use it, possibly even at the expense of making the app not be backwardly compatible with pre-Leopard versions of OS X.