Colorstream FAQ

When people see my color streams, the first question I always get is "What do you plan to do with them?"

Truth be told, I'm still trying to figure that out. I'm pursuing a few possible tracks though, and I haven't yet found the end of any of them:

- Mapping the sound spectrum "directly" onto the color spectrum, and seeing what it looks like.

- Making pretty pictures. Where the first track fails to be colorful, or interesting (which it so far hasn't), I'd like to tweak the algorithm to "see more" of the interesting aspects of the sound, and create interesting visuals.

- From nearly the start, it was my hope that I could take advantage of the eye's knack for pattern recognition, and use that to maybe recognize real speech and words, using color. Following this to the extreme, even to allow deaf people to see sound. I'm finding reality might stop this idea short of that end, but I'm seeing how far it goes anyway.

The next comment I always get is, "It would be great if it was a live stream, so you could see it while you heard it."

I had always pictured this as an eventuality for this project, or at least an ideal. These static strips of color are just approximations of the ideal which is probably a live leftward-scrolling color stream (the right hand edge being "exactly now"). Aside from this being beyond my current programming skills, for now I'm just developing the idea, and finding out what's possible.

* Image above: color stream of "color stream FAQ", spoken.

Lame-Arss-Gimp

It's still in development, but I'm finally "releasing" a usable tool to create color streams from sound files. It's downloadable here, as a humble zip file. Instructions for setup and use are included. It's really just a simple windows batch script that calls three open source programs in sequence (be it a coincidental and ironic sequence) to do the heavy lifting...

LAME: the script takes in a .wav or .mp3 file, and passes it into the LAME MP3 encoder, which outputs a clean .wav file.

ARSS: the .wav file is passed into ARSS (Analysis & Resynthesis Sound Spectrograph) with particular parameters which creates a spectrogram image.

GIMP: the spectrogram image is passed to a custom GIMP script (included), which applies the color map to the spectrogram, and does the color blending, and saves the final image to disk.

(I do not know how these programs got named. I disclaim responsibility.) If anyone would like to try out this tool, I'm interested in general feedback, and finding interesting examples of sounds to demo this thing. The color mapping (which still seems to be in flux) is currently optimized essentially for speech, but is all very hackable in its present state.

* Image above: color stream of "Lame-Arss-Gimp", spoken.

Colorstream: Sound into Light

The nice thing about my process for making these color streams (see previous post) is that it lends itself to visually illustrating each step -- with pretty pictures, no less.

We start with a sound file, and create a spectrogram out of it. If you looked at the first page of google results for "spectrogram", you would quickly decide they're only meant to be comprehended by math Ph.D.'s. (This is a shame, because spectrograms are so neat just on their own. There's a lot to them.) But in the end, they're as simple as this: the top of a spectrogram is the high tones (think mosquito in your ear), and the bottom is the low bass tones (think bass singer, singing his lowest note). Here is a spectrogram of me speaking the letters of my name "B-R-I-A-N". (mp3 file here.)



[Glossing over some of the fascinating and peculiar things about spectrograms for the time being*,]
Now, the low (bottom) pitches should map to the lowest frequency of light the eye can see: Red. The high pitches should map to the highest visible frequency of light: Blue/Violet. In between, should be all the colors of the rainbow (or the closest approximation a computer monitor can make).



Next, for each vertical slice (pixel) of the image above, we get the average color from bottom to top. Like mixing paint, this is what the eye would see if these colors were seen together. The progression below illustrates this "averaging" step.




Finally, we have a completed image, a stream of sound on the spectrum of visible light.



* For the mathematicians, the spectrogram above goes from 25 hz sound on the bottom to 10000 hz on top -- on, oddly, a log base 1.4 scale a scale blended between linear and logarithmic scale. More on this, in the future.

Colorstream project

You wouldn't know it from my online activity, but I've been working on a project to visualize sound. Basically, each moment in sound has a certain intensity of low rumbly noises (pitches), medium pitches, high hissy pitches, and all the pitches in between. In "white noise", all these pitches are equally on. It turns out light and color follow a similar pattern. There's a spectrum of color from red to blue (low pitch to high pitch), and a combination of a bunch of colors of the rainbow equates to a color we see with our eyes. All colors on looks white. All colors off, black.

Well, the basis of my project is, "What if we looked directly at the sound spectrum as if it were the light spectrum? What if we could see sound as color?" As it happens, when you look at the 20th Century Fox noise on the color spectrum instead of the sound spectrum, it looks like this (click for larger version):



Pee Wee Herman saying "Large Marge Sent Me" before some spooky music, like this:



The pictures above are the result of a bunch of experimentation to find a useful mapping from sound to color, and learning a few carefully selected programs involved in the process. The steps to do the conversion have been pretty manual, but I've been developing a tool that will automatically generate these "color streams". However, I'll get into that in a future post.

Dear Google

Dear Google,

Why won't you let me middle click the links in your "more" expand-o box?