Data sonification - turning information into sound and music

General
1

I’ve been promising to start a data sonification thread for a while, and so here it is. I thought it’d be fun to chat a little about the art and science of turning data of different kinds into sound and music - a process known as “data sonification”.

Sonification is way more common than you probably think it is. A checkout beep, an ECG monitor in a hospital, a geiger counter, sonar, a burglar alarm, a cuckoo clock, a custom ringtone. All of these are sonifications - conveying useful information or data in the form of sound. Speech doesn’t quality in most definitions of sonification, but that doesn’t mean you can’t use your voice as an intrument to make sonifications (I call this “punk sonification”).

In a lines-y context, I love using sonification as a way to add meaning and emotion to generative systems - why just throw random numbers into your quantizer when you could sample the cosmic weirdness and turbulence of the real world instead? There are datasets of global temperature, sunspots, species decline, stock prices, and much more that offer far more complex and nuanced sequencing or modulation than most generative approaches can offer - they come with a built-in story at the same time. There’s a bit more academic discussion about the philosophy and practice of sonification more broadly from @beo, @Net and @samarobryn in this thread from a few years back.

Over the last few years, one problem I’ve found is that it’s quite hard to create interesting sonification work without knowing how to write code, so I’ve invested a bit of time recently in building tools aimed at lowering that barrier to entry - including the Loud Numbers Norns script, and an upcoming VCV Rack module. There are also other sonification-related scripts in the Norns ecosystem - including @markeats’ Arp Index, @obi’s Fibonacci sequencer, perhaps @jaseknighter’s Flora (would you consider this sonification @jaseknighter?) and my own ppm script.

Also worthy of a mention is Brendan Byrne’s World Clock - a Eurorack module that generates clocks that match real-time birth rates in six different regions of the world (thanks to @infovore for clueing me into this) and of course the Plantwave device and Instruo Scion Eurorack module.

If you want to listen to some examples of what you can make with sonification, then you might start with our very own @rycolos’s A Home, Electric, @dennisoven’s NOKO210 work, anything in the Data Sonification Archive, or some of my work - including the Loud Numbers podcast, and London Under the Microscope.

In this thread, I’ve love to hear from folks who have experimented with sonification, or anyone who’s curious about its possibilities. Share your own sonification work, your favourite techniques and datasets, or anything else relevant to this growing, fascinating artform :chart_with_upwards_trend:

46 Likes
2

I wrote about the sonification.design website for the current issue of The Wire (the one with Reynols on the cover). I interviewed Sara Lenzi, who curates the website with Paolo Ciuccarelli. I’ll post the full text once the subsequent issue is out, in a couple weeks, I think.

20 Likes
3

I’m working on a norns script that reads plain text and turns it into pitch and trigger information to be sent out via crow. I think the idea is similar to what you’re describing above, particularly the way I’m implementing it right now, which essentially treats plain text as a dataset made of letters, converts those letters into a table of numbers, populates a sequin with those numbers, and uses that sequin to determine pitch and lattice patterns for timing. I’ve got it working in the way described above, though I think it’s far from releasable. For one thing, it doesn’t have any sort of GUI yet. But I’m happy to share the script with anyone who’s interested!

Eventually, I hope to create a version of this script that can be more user-friendly, possibly to create an installation piece with a small lunchbox-type eurorack setup, a 16n faderbank, a norns shield, and a qwerty keyboard people could type words on to create music. But that’s a long way off. Another area I’m interested in exploring is natural language processing, perhaps taking advantage of norns’s wifi connectivity to somehow use existing online tools for affect analysis that could determine quantization settings. But I don’t even really know where to begin with that. Just thought I would share here because the ideas I’ve been playing around with seemed relevant to the topic. Looking forward to seeing what others have to share here!

6 Likes
4

Sara and Paolo are top folks, doing great work! Hoping to partner with them on a sonification community project in the coming months.

This sounds really cool! It’d be so fun to feed it a short story and get a song out. Maybe the visual interface should be the script typing out the text, character by character, as the sound plays?

Automatically quantising to different scales based on text sentiment analysis is really interesting too. Such good ideas!

2 Likes
5

yes! I think this would be the most satisfying way to approach a visual representation of the script. Ultimately, I think I’ll wind up with separate versions, one for loading in an existing text (e.g., “I wonder what this poem I just read would sound like as music?”) and one for inputting text through a keyboard for the aforementioned installation art context. I’m imagining the latter looking something like the GUI for the crow talk script (which is to say, something like a command prompt where previous input will appear above the “command line”).

For the moment, though, until I have time to dive into learning how to create a norns GUI, I’ve been treating the script’s “invisibility” as an advantage and creating combo scripts with it. I did my first poetry reading with this setup a few weeks ago, using my script in a combo with mlr so I could cut up and manipulate the sound of my voice reading poems while the synthesizer played the poems I was reading. I’ve also been having fun feeding the script other people’s poems and using mlr to cut up recordings of those poems being read by their authors. The PennSound archive has been a great resource for this. Here’s an example.

Quantization has been very helpful for making these experiments more listenable, but more recently, I’ve been playing around with turning quantization off and using the script as a compositional tool for poetry (i.e., trying to write poems that will make pleasant and/or interesting music).

Sentiment analysis was the term I was thinking of! Thanks for this. But also, my not remembering the term shows how much I know about this stuff, which is to say, pretty much nothing other than that such tools exist. If you’ve got any ideas for how to incorporate something like this, I’d love to hear them and explore! I think that could really take the script to the next level.

I’ve also got another problem that I thought you or someone else might have some ideas about. Right now, I’ve got the script turning text into numbers by splitting the text string on spaces. Unfortunately, that means I have to go through and manually insert spaces between each character of the text. Not doing so, I’ve found out the hard way, crashes norns. So what I wind up with is something like this (the text is from me playing around with letters that I thought would make interesting music, as mentioned above):

title = "t i r e b y t e"
text = "g o t r i g h t g o t r i g h t g o t r i g h t g o t r i g h t g i t w o t i t t o w l g o t t o w l g o t w i t r i g h t w r i t o w t w o t w r i t i t r i g h t w r i t o w t w r i t o w t w r i t o w t w r i t o w t w o t w r i t i t r i g h t"

Each of the letters in the string gets replaced with a number using string.gsub. This string of numbers then goes through a splitter to turn into a table, like so

function Split(s, delimiter)
  result = {};
  for match in (s..delimiter):gmatch("(.-)"..delimiter) do
    table.insert(result, match);
  end
  return result;
end

lied = Split(Znumbs, "%s")

I feel like there should be a way to feed the script a text and get it to split it up on its own, without having to manually insert those spaces. Ideally, a user could just copy a .txt file over to the script folder, the same way Loud Numbers uses .csv files, and the script would figure out the rest. Any ideas for how I might go about that? Right now, I don’t even know how to direct the script to look for a file in the folder, which I suppose would be the first step.

By the way, I’ve been listening to the Loud Numbers podcast today, and it’s great! What a fun idea for a show.

6

Great you are featuring data sonification as a compositional tool.
I mainly use the ChucK coding environment as my go to for implementing these things although there are obviously a myriad of coding languages, hardware platforms and methods right down to pencil and paper.

Scott.

1 Like
7

I’d never heard of ChucK before! This looks really interesting. Thanks for mentioning it

8

wow, just imagining a future where a soundtrack to a novel is generated in real time as you read!

3 Likes
9

Maybe you could do something like:

local str = "text"
local t = {}
for i = 1, #str do
    t[i] = str:sub(i, i)
end

I’d definitely recommend getting yourself super familiar with the lua string library as a first step. There’s a lot you’ll be able to do just with that, I think. Make sure to test with multibyte characters :slight_smile:

For sentiment analysis, it looks like IBM Watson’s Natural Language Understanding model has a fairly generous free tier, but I’ve never used it so I’ve got no idea how good it is!

Feel free to steal anything you like from the Loud Numbers script! My “list file names” function is here, for example. Note that I’m using an external library to parse the CSVs, but you probably won’t need to do that with plain text.

Thankyou so much for the kind words :heart_eyes:

2 Likes
10

In my current project I use Flora and a biosonification device. What interests me is not to get midi/cv pitch, scale etc information from the biodata but frequencies. Wouldn’t this widen the sonic spectrum?

1 Like
11

I would recommend checking out some of what Carla Scaletti has written/said on this topic.

I have the “sonification≠music” chapter available as a PDF—if anyone wants to dig into it, just message me.

6 Likes
12

@radioedit, don’t think i’ve ever considered flora a piece of data sonification (that’s not to say it isn’t).

my approach in flora (and the new krill script i just published) was to take a visual form generated algorithmically and figure out how that form can be translated into something musical. in the case of flora, the starting point was to turn changes in the angles of the plant forms into changes in pitch. in the case of krill, the starting point was to map the drawing of the lorenz shape to a grid that could be used to set pitch and octave.

i’m curious about music that is played from a musical score. is that considered data sonification?

3 Likes
13

So I have a particular perspective on this (one colored by my academic training and artistic outlook, YMMV).

Firstly, data sonification does not yield “music” (so our thread title may be more relaxed than my outlook, and perhaps a tad misleading). Music, sonification, and language are three separate and distinct ways to convey various kinds of meanings (with varying degrees of specificity and, some may say, success-rates) through sound. Sonification in particular attempts to use sound to convey information about scientific data. It is not concerned with what Scaletti calls “the auditory sublime” or conveying affekt, emotion, expression, culture.

A score is not data; it was not collected, measured, observed; it consists of (generally arbitrary) choices made by a composer in order for someone else to perform it (rendering it audible through sound). But again music doesn’t simply “sonify” the score—there is a wide-ranging debate about “where” the music actually is (in performance? On the page? inside the composer’s head? Somewhere in between? All of the above?) as well as how well music actually “communicates” anything (if anyone tells you they know what a piece of music “really means”, don’t trust them!).

Of course we live in postmodern times and people often put forth their own definitions of things—I’m reporting from what I understand is the general professional/academic consensus around these terms (with the caveat that “sonification” is not really a widely-understood concept, nor a word that appears in most dictionaries!)

8 Likes
14

love the resonator-like nature and the general fm-iness of this

3 Likes
15

Loving this discussion! Thankyou @corbetta and @jaseknighter for bringing up great questions. I will try to give my answer to them, and I would love to hear other perspectives on the subject from other community members. All of the below should be prefaced by the fact that I have never studied the philosophy of music in an academic context, and that I’m very open to having my mind changed. I love exploring this stuff, and it’s not the kind of discussion I get to have in my daily life very often.

I definitely have a much broader, more inclusive definition of sonification than the academics do.

I consider it to refer to the process of turning information systematically into sound. That usually means the system of decisions and choices you make when converting something that isn’t a sound (e.g. a musical score, a feeling or emotion, a table of numbers) into a sound, and data sonification is when the input to that process is data (of all kinds, not just “scientific” - which sounds concerningly exclusionary to me - I am a data humanist).

So yes - I would consider performing music from a score to be a sonification process, in the sense that the score contains information, and when you play from it you’re sonifying that information. The fact that different people can take the same input (a score) and create the same ouput (a sound) but get totally different results implies there must be an interesting process in the middle. I call that process sonification.

I hope the above explains why I would consider that approach to be sonification - you’re making decisions on how to represent the algorithm/visual with sound. But to reiterate, this is very much just my perspective and I would love to hear the perspectives of others :slight_smile:

6 Likes
16

Thanks @radioedit —I’m continuing the discussion in the spirit you’re suggesting, not necessarily trying to correct you or change your mind.

My issue with that outlook is, I guess, categorical. Score reading is one approach to music making that is prevalent (though not as much as it used to) in western musics, but there is obviously a lot MORE musicking that happens, and has happened, without notation. So I can’t really see it as “a kind of sonification” because that seems sort of backwards to me—it’s just making music in a particular, and definitely not exhaustive way. The notation is in some way “information” but it’s perhaps better described as either “instructions” (of a sort) or a “trace” of the work; music is already sound and as such cannot be “sonified” (with this last statement I may have officially entered the realm of sophism). Scores are especially complicated because a strong reader will effectively “hear” the music within their heads just by glancing at the score—not that I know that from direct experience, mind you. But the “essence” of the musical work (within the context of notated music) is really hard to locate with precision.

Is reading a book out loud sonification? What about speaking?

The other way I’m looking at this is more in line with Scaletti and has to do with intent; sonification renders audible “information” that wouldn’t otherwise be audible, in an attempt to discover, highlight, or convey meaningful patterns and relationships within. In my world music doesn’t do that—just like music and language, music and sonification are related, contiguous, but ultimately separate domains.

6 Likes
17

could we say, then, that performing music from a score is a sonification of the trace, which is not, in itself, music but which participates in and is related to music?

Is the data not the score in this sense? It could be understood as a trace (i.e., temperature data are not temperatures, they are a trace left by the measurement of temperatures) which can then become instructions to be interpreted by, as @radioedit put it,

In this sense, a score is a trace in the same way temperature data is a trace, but whereas temperature data is a trace left by temperatures, which can then be turned into music, a score is a trace left by music, which can then be turned into music.

Part of what I find interesting about this whole idea of “sonification” is that it changes the kinds of decisions we make as interpreters. In my experience with the script I mentioned above, my attention is initially drawn to how I’m designing the system (i.e., how am I mapping letters in plain text to notes) and then to the timbres of the destination instrument (in this case, a eurorack setup, since I’m using crow).

1 Like
18

Some very interesting ideas here glazing my brain at work. I just wanted to write this down while I still remember, but having brushed with @radioedit new Loud Numbers VCV module I think .CSV must have carved out small alcove in my brain and I couldn’t help noticing that NYSTHI has a module that lets you create CV generated .CSV files. It’s called LOGAN
Seems like there’s some fun to be had here :smiley:

4 Likes
19

I shared this on Discord already but it belongs here, too: an excerpt from Douglas Adams’ Dirk Gently’s Holistic Detective Agency:

image
image1920×2433 431 KB

image
image1920×2994 531 KB

image
image1920×823 153 KB

12 Likes
20

I mean, data can become a score through artistic choices, but sonification in the scientific/communication world is not artistic. The choices made to sonify the data are not done following aesthetics, but intelligibility and clarity. In other words, I would say that if you’re making artistic decisions then you’re composing, not sonifying.

There is definitely some room for an overlapping field in terms of “data-generated composition” but again I would want to give it a different name. Maybe “musical sonification” (though again that’s kind of redundant), like a sort of “sound poetry” hybrid. Or then again, maybe it’s just “composition.”

2 Likes