Data sonification - turning information into sound and music

I’d never heard of ChucK before! This looks really interesting. Thanks for mentioning it

wow, just imagining a future where a soundtrack to a novel is generated in real time as you read!

3 Likes

Maybe you could do something like:

local str = "text"
local t = {}
for i = 1, #str do
    t[i] = str:sub(i, i)
end

I’d definitely recommend getting yourself super familiar with the lua string library as a first step. There’s a lot you’ll be able to do just with that, I think. Make sure to test with multibyte characters :slight_smile:

For sentiment analysis, it looks like IBM Watson’s Natural Language Understanding model has a fairly generous free tier, but I’ve never used it so I’ve got no idea how good it is!

Feel free to steal anything you like from the Loud Numbers script! My “list file names” function is here, for example. Note that I’m using an external library to parse the CSVs, but you probably won’t need to do that with plain text.

Thankyou so much for the kind words :heart_eyes:

1 Like

In my current project I use Flora and a biosonification device. What interests me is not to get midi/cv pitch, scale etc information from the biodata but frequencies. Wouldn’t this widen the sonic spectrum?

I would recommend checking out some of what Carla Scaletti has written/said on this topic.

I have the “sonification≠music” chapter available as a PDF—if anyone wants to dig into it, just message me.

4 Likes

@radioedit, don’t think i’ve ever considered flora a piece of data sonification (that’s not to say it isn’t).

my approach in flora (and the new krill script i just published) was to take a visual form generated algorithmically and figure out how that form can be translated into something musical. in the case of flora, the starting point was to turn changes in the angles of the plant forms into changes in pitch. in the case of krill, the starting point was to map the drawing of the lorenz shape to a grid that could be used to set pitch and octave.

i’m curious about music that is played from a musical score. is that considered data sonification?

3 Likes

So I have a particular perspective on this (one colored by my academic training and artistic outlook, YMMV).

Firstly, data sonification does not yield “music” (so our thread title may be more relaxed than my outlook, and perhaps a tad misleading). Music, sonification, and language are three separate and distinct ways to convey various kinds of meanings (with varying degrees of specificity and, some may say, success-rates) through sound. Sonification in particular attempts to use sound to convey information about scientific data. It is not concerned with what Scaletti calls “the auditory sublime” or conveying affekt, emotion, expression, culture.

A score is not data; it was not collected, measured, observed; it consists of (generally arbitrary) choices made by a composer in order for someone else to perform it (rendering it audible through sound). But again music doesn’t simply “sonify” the score—there is a wide-ranging debate about “where” the music actually is (in performance? On the page? inside the composer’s head? Somewhere in between? All of the above?) as well as how well music actually “communicates” anything (if anyone tells you they know what a piece of music “really means”, don’t trust them!).

Of course we live in postmodern times and people often put forth their own definitions of things—I’m reporting from what I understand is the general professional/academic consensus around these terms (with the caveat that “sonification” is not really a widely-understood concept, nor a word that appears in most dictionaries!)

7 Likes

love the resonator-like nature and the general fm-iness of this

3 Likes

Loving this discussion! Thankyou @corbetta and @jaseknighter for bringing up great questions. I will try to give my answer to them, and I would love to hear other perspectives on the subject from other community members. All of the below should be prefaced by the fact that I have never studied the philosophy of music in an academic context, and that I’m very open to having my mind changed. I love exploring this stuff, and it’s not the kind of discussion I get to have in my daily life very often.

I definitely have a much broader, more inclusive definition of sonification than the academics do.

I consider it to refer to the process of turning information systematically into sound. That usually means the system of decisions and choices you make when converting something that isn’t a sound (e.g. a musical score, a feeling or emotion, a table of numbers) into a sound, and data sonification is when the input to that process is data (of all kinds, not just “scientific” - which sounds concerningly exclusionary to me - I am a data humanist).

So yes - I would consider performing music from a score to be a sonification process, in the sense that the score contains information, and when you play from it you’re sonifying that information. The fact that different people can take the same input (a score) and create the same ouput (a sound) but get totally different results implies there must be an interesting process in the middle. I call that process sonification.

I hope the above explains why I would consider that approach to be sonification - you’re making decisions on how to represent the algorithm/visual with sound. But to reiterate, this is very much just my perspective and I would love to hear the perspectives of others :slight_smile:

6 Likes

Thanks @radioedit —I’m continuing the discussion in the spirit you’re suggesting, not necessarily trying to correct you or change your mind.

My issue with that outlook is, I guess, categorical. Score reading is one approach to music making that is prevalent (though not as much as it used to) in western musics, but there is obviously a lot MORE musicking that happens, and has happened, without notation. So I can’t really see it as “a kind of sonification” because that seems sort of backwards to me—it’s just making music in a particular, and definitely not exhaustive way. The notation is in some way “information” but it’s perhaps better described as either “instructions” (of a sort) or a “trace” of the work; music is already sound and as such cannot be “sonified” (with this last statement I may have officially entered the realm of sophism). Scores are especially complicated because a strong reader will effectively “hear” the music within their heads just by glancing at the score—not that I know that from direct experience, mind you. But the “essence” of the musical work (within the context of notated music) is really hard to locate with precision.

Is reading a book out loud sonification? What about speaking?

The other way I’m looking at this is more in line with Scaletti and has to do with intent; sonification renders audible “information” that wouldn’t otherwise be audible, in an attempt to discover, highlight, or convey meaningful patterns and relationships within. In my world music doesn’t do that—just like music and language, music and sonification are related, contiguous, but ultimately separate domains.

5 Likes

could we say, then, that performing music from a score is a sonification of the trace, which is not, in itself, music but which participates in and is related to music?

Is the data not the score in this sense? It could be understood as a trace (i.e., temperature data are not temperatures, they are a trace left by the measurement of temperatures) which can then become instructions to be interpreted by, as @radioedit put it,

In this sense, a score is a trace in the same way temperature data is a trace, but whereas temperature data is a trace left by temperatures, which can then be turned into music, a score is a trace left by music, which can then be turned into music.

Part of what I find interesting about this whole idea of “sonification” is that it changes the kinds of decisions we make as interpreters. In my experience with the script I mentioned above, my attention is initially drawn to how I’m designing the system (i.e., how am I mapping letters in plain text to notes) and then to the timbres of the destination instrument (in this case, a eurorack setup, since I’m using crow).

1 Like

Some very interesting ideas here glazing my brain at work. I just wanted to write this down while I still remember, but having brushed with @radioedit new Loud Numbers VCV module I think .CSV must have carved out small alcove in my brain and I couldn’t help noticing that NYSTHI has a module that lets you create CV generated .CSV files. It’s called LOGAN
Seems like there’s some fun to be had here :smiley:

4 Likes

I shared this on Discord already but it belongs here, too: an excerpt from Douglas Adams’ Dirk Gently’s Holistic Detective Agency:



7 Likes

I mean, data can become a score through artistic choices, but sonification in the scientific/communication world is not artistic. The choices made to sonify the data are not done following aesthetics, but intelligibility and clarity. In other words, I would say that if you’re making artistic decisions then you’re composing, not sonifying.

There is definitely some room for an overlapping field in terms of “data-generated composition” but again I would want to give it a different name. Maybe “musical sonification” (though again that’s kind of redundant), like a sort of “sound poetry” hybrid. Or then again, maybe it’s just “composition.”

2 Likes

Googling for “supercollider sonification” proves to be productive.

Reading the above as well

:chart_with_upwards_trend::chart_with_downwards_trend::chart_with_upwards_trend::chart_with_downwards_trend::chart_with_upwards_trend:
:chart_with_upwards_trend::chart_with_downwards_trend::chart_with_upwards_trend::chart_with_downwards_trend::chart_with_upwards_trend:
:chart_with_upwards_trend::chart_with_downwards_trend::ear::chart_with_downwards_trend::chart_with_upwards_trend:
:chart_with_upwards_trend::chart_with_downwards_trend::chart_with_upwards_trend::chart_with_downwards_trend::chart_with_upwards_trend:
:chart_with_upwards_trend::chart_with_downwards_trend::chart_with_upwards_trend::chart_with_downwards_trend::chart_with_upwards_trend:

2 Likes

Thank you for starting this thread! Really looking forward to how this unfolds.

I work for an academic library and few colleagues and I started an extracurricular project to sonify the metadata describing scholarly literature, citations. It is the most slow motion (cuz extracurricular) project I’ve ever been involved in, so I don’t have a lot to show for at the moment. We also read the Scaletti article/chapter “Data Sonification ≠ Music.” Like @corbetta, I would strongly recommend it and I would consider myself to be the resident Scaletti-ist of our project, though that puts me in the minority.

Just a few quick thoughts until I have more time to share:

TL;DR: auditory semantic literacy is not nearly as well established as the visual equivalent and there are some surprising gotchas.

I would just like to +1 @corbetta’s comment that the distinguishing feature is the “mapping” and “intention” aspect to sonification. They go hand in hand in terms of understanding that perspective and it helps me understand the distinction between, for example, temperature data and a score. In some sense the notes in a score might be already equivalent to MIDI note numbers (of course, I realize there are problems with that generalization!) through their intended use and that they don’t get mapped in quite the same way that temperature data does. My example here would simply be that the mapping for temp data to some musical parameter to, say, the 0-127 MIDI range is more of a mapping because the sonifier has to make choices like, “What range of temperatures should be mapped to what range of the MIDI integers?” And once you take into account things like the logarithmic nature of the human frequency experience, that temperature data mapping might involve a lot more mapping, as it were, than a score.

One of the things that I have constantly struggled with on our project is that the most obvious examples (in terms of what members of the group have suggested most frequently) usually involve mapping some numerical data element to pitch but that this can frequently be the opposite of the psychology of human hearing. Or at least it makes for surprising results that don’t convey meaning, which, again, if one is a Scaletti-ist :slight_smile: , is a core purpose of data sonification analogous to data visualizations like bar charts, line graphs and scatterplots. Large numbers mean in their simplest sense a higher quanity of X. However, very high frequencies often are experienced as “tinny” or “thin” sounding and have conveyed small-ness in some of our experiments. Lower numbers mean less of X, but often my animal brain hears a low frequency rumble and experiences a swell, some large-ness.

Another way that I have struggled with the ways where the psychology of hearing is reversed from simplistic numerical meaning has to do with pitch proximity. We use data visualization as the more established practice/field as a kind of metaphor in our project to assess the effectiveness of communicating meaning. But the visual field works well with many kinds of intuitive numerical meanings. For example, on a scatterplot, two points with number pairs that are near each other also have X/Y number pairs that are near each other. In both the visual representation and the numerical representation, we can convey closeness/clustering and therefore convey similarity. Now when we try nearness with pitch, the auditory experience is the opposite, there is no consonance that conveys similarity. Two pitches that are close, but not the same, by their simple numerical frequency representation sound dissonant. In our group this tends to produce an interpreted experience of the data that suggests things are dissimilar, which is not what we wanted.

9 Likes

I am now officially fascinated. Could you send me the pdf? After looking at Scarletti’s website, I’m dying to read it

Great to read your thoughts, @stephenmeyer — the issues you bring up about pitch and perception are close to my interest areas (I’m a tuning nut/advocate). I’d add octave equivalence (or lack thereof) and pitch quantization as crucial elements that are often glossed over and/or give rise to complications on both the “artistic” and “scientific” side of the practice (for lack of better terms). But there’s much more to sound than pitch!

Don’t know if this is an ongoing concern but perhaps looking at something like Euler’s Gradus Suavitatis or Tenney’s Harmonic Distance function (in a justly-tuned context) could be used to convey “proximity” and “distance”?

3 Likes

Thank you for these recommendations. I will look into them and if you have any suggested citations for good intros, I will track them down through my library! I have neither formal music nor math training, so these things do take a while for me to internalize. Often worth the effort, tho!

This. Many times over! I’d be curious if anyone here has knows of a sonification vocabulary, as it were. Guidance, such as, use timbre as a categorical indicator equivalent to color/shape in data visualization, ___ as magnitude, etc.

2 Likes

What a great discussion this is turning into. Thankyou to everyone for participating :green_heart:

Yeah, I think that’s the key difference between our viewpoints. In my world, music does do that, and so I don’t feel any need to separate separate sonification and music into different categories.

I actually really love the idea of a “trace” that you mention. Information is a “trace” of something that happened in the world, and you can turn that trace back into an experience again in many different ways. If you turn it into sound, then like @WilliamHazard I would call that sonification.

This sounds a little like a sonification version of Edward Tufte’s arguments that anything that isn’t pure information transfer in a visualization is “chartjunk”. Those ideas have their place when your goal is solely to communicate information to a receptive audience as efficiently as possible. They’re less effective when your goal is to grab attention, charm, persuade, or spark emotion.

When I’m creating music with data (in my podcast, for example), I’m usually trying to one of those latter things. So when I’m making the decisions about how to map the data to sound (which is what I use the word sonification to refer to), I’m balancing up: “does this effectively tell the story?”, “does this represent the data accurately?” and “does this sound good?”.

Yes!! This is something we encountered a lot while working on the podcast. Pitch is often the first mapping that people reach for (sometime it’s volume), but it’s rarely a good choice for many of the reasons that you both outline. I find it most useful when it’s limited to a relatively narrow range, mapped inversely to how you might expect (so lower pitches are bigger numbers) and used to convey very small changes, rather than very large ones. Even then, the issues that you bring up around consonance and dissonance are still an issue (see the discussion in this post).

These days I try to use pitch for category information instead, like in my London Under the Microscope sonification, where different variants of the virus are represented by different notes within a chord, fading in and out as they rise and fall in prominence.

Another little trick that I used in that piece is to use a secondary mapping to handle outliers. Cases and deaths are primarily mapped to volume/amplitude, and when the data is within “normal” range, it has a strong LPF on it. But when they spike up out of the “normal” range, the filter opens up and you hear the higher frequencies, making it sound more “raw” and emphasising that this is weird and unusual.

I’ve heard this request many times, but sonification is in its infancy compared to visualization and so I haven’t seen one pop up yet. Maybe we should try to collaboratively compile something? I think the objective should probably be more in the vein of “advice/good starting point/this has worked for me”, rather than “this is objectively the best approach”.


One last thing, today I spotted Ableton plugging sonification as a compositional technique in their “One Thing” series :heart_eyes:

4 Likes