Mapping Music to Color

[keefo]

Hi All

I am looking for good idea to map music to color. If you have something to share, please join this discussion.

Regards,
LX

[photonclock]

What are you looking for particularly? A user interface approach and how the light would change in relation to music?
Ideas:
A hue scale: ie [red...yellow...green...cyan...blue]
A brightness slider ie 0-100
A threshold/sensitivity slider ie 0-100
User could set a couple of markers on the hue slider to indicate hue range (ie, quiet to loud = yellow to cyan)
Brightness slider would set max brightness correlating to max audio level
Threshold slider would scale overall sensitivity to the audio input level
The hue slider could have a 0-360 control to rotate the hue range across the scale so the user could adjust the start/end colors, ie,
[red...yellow...green...cyan...blue]
[...yellow...green...cyan...blue...red]
[...green...cyan...blue...red...yellow]
Hue slider could also have an "invert" checkbox to reverse the color range

[keefo]

I am looking for something more sophisticated. Something like NLP sentiment analysis, it can detect the sentiment of a music, and map for example a "sad" music to "blue" light. etc.

[keefo]

It doesn't have to be sentimental stuff. Another example, map strong beats during a music to a flash.

[ffs889]

Hey @keefo,

Mapping strong beats to a flash is fairly straightforward with a Fast Fourier Transform (FFT). That will help you break audio into frequencies (bass, mid, treble).

As a starting point
You can use AudioKit or the Accelerate Framework for basic frequency analysis.

You can then map bass to warmer colors (red, orange), midrange to mixed colors (yellow, green) and higher frequencies to cooler colors (blue, purple), using the amplitude of frequencies for the brightness levels.

Better
You could use the (free!) Spotify API to fetch the 'valence' and 'danceabiliity' of a song, and map those to different color profiles. See: Audio Features endpoint.

e.g. low valence + low danceability could be closer to blue, high valence + high danceability lots of orange and yellow.

Sentiment mapping, you say?
Take a look at Spotify's (free!) Audio Analysis endpoint.

Here, each song is broken up into distinct "sections" (defined by large variations in rhythm or timbre, e.g. chorus, verse, bridge, guitar solo).

Also check out this Python app, which uses that endpoint to match tempo, pitches and loudness to RGB outputs. You'll find beautiful visual timelines of the RGB output at the bottom.

AudioKit also has tools for beat and tempo detection, pitch tracking, and timbre analysis, but it seems to me that the Spotify API already did the heavy lifting.

I'm glad to have found this project, and I would love to see this feature come to life!

[ffs889]

See also this web app which uses Spotify's Audio Analysis API to control Philips Wiz lights (RGB), specifically its "dance engine" and color picker.

[keefo]

We have a new powerful Sound to Light Engine now.

https://github.com/keefo/NeewerLite/blob/main/Docs/Sound-to-Light-Engine.md

Tech report

https://github.com/keefo/NeewerLite/blob/main/Docs/Sound-to-Light-Technical-Report.md