Chromasthetic Fairies

In this performance, I intended to create an ambient atmosphere which explores the interplay between colors and ambient sonic textures. It’s a pretty rough attempt to try and recreate the experiences I’ve had with chromasthesia. As to how I managed to get those experiences, I cannot tell you unfortunately.

Each of the different circles represents set of frequency bins that correspond to one of the 12 pitches of the western equal temperament system. As the program takes in audio signals, chroma features are extracted out, and each circle gets activated on a continuum depending on how active its respective chroma is.

I decided to get rid of the thresholding from the previous milesone. This was because it added alot more visual activity and volume to the collection of circles, and to help accomodate the fact that I was now playing chords. For each of the chroma, I used a sigmoid function that mapped from chroma values to the range from 0 to 1. To get the visualizer to look consistent, I needed to calculate an exponential moving average for each of the chroma. This solved a problem I had in previous milestones where each of the different chroma would be finnicky, even if an audio signal persisted.

After a few seconds, the circles start roaming around randomly. There is no real driver for the paths the circles take, it’s just random. This was added to make this more visually engaging.

Overall, I’m quite happy with what I created, given the time frame and amount of free time available to work on this.

Code is here. Unzip it and run chuck chromasthetic to have this respond to the microphone. If you want to see the demo, run chuck chromasthetic:demo.

Milesone 3

Milestone 2

This is the current prototype of what I want to ultimately build:

As one can see, this is very similar to the sonic visualizers that we worked on at the beginning of the course. Here, however, the difference is that it isn’t necessarily responding to raw audio waveforms or their STFT: this is reacting to an audio signal’s chroma features. Each of the evenly-spaced circles correspond to different pitch chroma – that’s why each of their colors are different. As an audio signal’s chroma features are analyzed, each of the different circles radiates out a torus with an intensity corresponding to the assigned chroma value’s strength.

At its core, I want this project to explore the concept of synesthesia, which is a condition where multiple senses are experienced simultaneously. One of the manifestations of this phenomenon is chromasthesia, where people experience associations between colors and sound. For this reason, I visualize each of the semitones in the western 12 equal temperament system as its own color. As you can see, C is blue. This is because I perceive C as a blue note.

One of the ideas that I was discussing with Andrew was the possibility of using fragment shaders to turn the whole screen into a fluid simulation, and in place of these radiating circles, fluid is advected out into existence, achieving a similar effect. 12 different locations of outward fluid advection would advect fluid of different colors, similar to the different colors in this example. This is going to be the goal for me, and I will be in talks with Andrew to see how this can be achieved.

Here are the questions I have on how I can move forward with this project:

  1. As it stands currently, the core mechanic of my project as of this moment is the visualization of these chroma; however, I am wondering how I could flesh out this visualization so that I can enable more interactivity between the user and the chroma features that are extracted, so I am more than willing to take any ideas from the audience on this. In other words, what sorts of other interactions would you all like to see?

  2. As it can be seen in the demo, sometimes, when I play a specific note, other pitch chroma that don’t correspond to that note are activated. This is also seen in the chroma.ck example in the ChucK page, where a sine oscillators’s pitch corresponds to a very specific note and sorrounding pitch chroma are still activated. How can I make it so that these unrelated pitch chroma are attenuated further? One idea would be to just take the max and argmax of the entire chroma vector, but this means that I can’t have multiple chroma activated at the same time when I play chords. Any ideas on how to better leave these undesirable chroma out is appreciated

Milestone 1

Idea 1:

Fluid simulation that is responsive to Chromagram features, running on the GPU via shaders. These chromagram features correspond to specific areas in the place

Idea 2:

Using RAVE for live audiovisual performance. The latent space directly drives geometric visuals. This can be driven by either the keyboard or audio input.

Idea 3:

Using a chaotic system for FM synthesis. Like a chaotic pendulum that is able to generate audio, and this audio has structure. The timbre is determined by the chaotic properties of the system.

Sketches: