Artificial Intelligence in Music

holly herndon and her partner have been cooking up a neural network that created the song that herndon just released, called “Grandmother.” here is a note on the process, from NPR:

“Godmother” was generated by listening to Jlin, Herndon’s friend and spiritual sister in unclassifiable electronic music, and then reinterpreting the data in Herndon’s voice. There’s a raw, newborn quality to the track as it hums and sputters like a swarm of glitching bees, just trying to find its mother.

and here’s part of herndon’s statement on the AI:

Going through this process has brought about interesting questions about the future of music. The advent of sampling raised many concerns about the ethical use of material created by others, but the era of machine legible culture accelerates and abstracts that conversation. Simply through witnessing music, Spawn is already pretty good at learning to recreate signature composition styles or vocal characters, and will only get better, sufficient that anyone collaborating with her might be able to mimic the work of, or communicate through the voice of, another.

Are we to recoil from these developments, and place limitations on the ability for non-human entities like Spawn to witness things that we want to protect? Is permission-less mimicry the logical end point of a data-driven new musical ecosystem surgically tailored to give people more of what they like, with less and less emphasis on the provenance, or identity, of an idea? Or is there a more beautiful, symbiotic path of machine/human collaboration, owing to the legacies of pioneers like George Lewis, that view these developments as an opportunity to reconsider who we are and dream up new ways of creating and organizing accordingly.

I find something hopeful about the roughness of this piece of music. Amidst a lot of misleading AI hype, it communicates something honest about the state of this technology; it is still a baby. It is important to be cautious that we are not raising a monster.

the full statement: Holly Herndon🧫 on Instagram: "Statement on @protospawn . Love to godparents Jlin & Jules LaPlace for helping train her so well 🤰"

on one level, i am disturbed by this eerie interpretation of a person’s voice by a neural network and the suggestion that the program can interpret and copy human compositional styles. on another, this kinda just seems like generative music, now with the buzzy phrase “AI” included (which might be a misinterpretation on my part).

what does everyone think about this?


Disclaimer: I am no expert w/r/t data science, and yet I still must opine…
I think all these machine learning techniques are purposefully and deceptively named with a lexicon designed to relate to words typically used to describe human intelligence. “AI” in this context has nothing to do with human intelligence - it’s classification technology, using some arbitrary system to improve what is probably a surprisingly simple model. I think the results still sound fascinating, but the way this kind of work is often presented obscures what is actually being done behind the scenes - what data is being used to train the model and how the output sound is actually being synthesized.

I think it’s disgraceful that “AI” is still being described within the science fiction context of a machine which can think like a human (or even an animal). The danger of “AI” is not replacing people - it’s management exerting more power over labor, it’s governments prioritizing bureaucracy and cost efficiency over humans, etc.

There are so many people out there misinforming us about what “AI” is, let alone how it’s being used already. Unfortunately, I feel like Holly Herndon is one of them, even if I respect her as an artist.

And of course Dryhurst is involved, the same snake oil salesman who spent years telling everyone how crypto coins would save musicians! I think the real issue being revealed here is not that of originality in a world of machine mimicry - it’s about how to generate the PR to be a successful musician when human mimicry is already so commonplace.


I think it’s totally fair to categorize ML-generated or -assisted music under “generative music.” The interesting differences, to me, are that (a) the initial step of assembling a corpus of training data can be super interesting – it’s a critical creative input – and (b) the network of “rules” the system learns is so complex its ends up being quite mysterious or even opaque. Which is interesting!

Sound from these systems also tends to have a particular “character” right now, which I think you can hear in the Herndon track. Most (all?) of the ML methods for generating raw waveforms operate on 16 kHz data. We’ll get to higher sample rates eventually, but for now (a) 16 kHz seems to give you the right trade-off of moment-to-moment quality to long-range structure, and (b) a lot of these ML techniques were initially designed for voice applications, for which 16 kHz is fine. As a result, in addition to all the other interesting artifacts introduced by these techniques, there’s a cloak of grainy, tape-y, 16 kHz quality. I happen to really like it!

You can hear some examples of what I mean in the audio players halfway down this page:


at first I was like “wow they made a lot of cool edits to the AI voice”

but nope, the ML shit that whole thing out, audio and all. pretty good.

Edit: this


great stuff, i would be interested in using similar tech to do arrangements, because it’s simply too boring for my artistic soul :wink:

Every implementation of crypto currency I’ve seen works by essentially regulating finance through pure supply and demand, mediated by a sophisticated technological infrastructure - this hardly seems “distributed” to me; it sounds like an even more unforgiving version of what exists now. The institutions we have to regulate finance are far from perfect, and they often benefit powerful individuals and organizations, but I would argue that the adoption of a highly technical and Hayekian economic system would be far worse. At the end of the day, these institutions (central banks and governments) in some way must respond to political pressure, whereas crypto is mediated purely by supply, demand and technology. How is it that a financial scheme which can literally only exist with the technology produced by handful of powerful corporations will empower individuals and “decentralize” anything? How is that “anti-computational”? If you’re naive enough to assume that the only villains in the world are banks and governments, I guess that’s enough. I really cannot see any realistic use case for blockchain that does anything but reinforce existing power structures. “Smart contracts” to me sound like an extreme libertarian fantasy of a world without trust or recourse. Western society has used the myth of individual empowerment and freedom for centuries to justify its course… which in my opinion often leads to perverse and unexpected outcomes. I see crypto as an unnecessarily technocratic “solution” to the wrong problem - an ultimate realization of a world in which society has been utterly replaced by transactions.

I don’t and can’t know that either Herndon or Dryhurst’s points of view are “wrong”, but they rub me the wrong way. The argument for AI as a collaborative, “symbiotic” entity is the same kind of rhetorical nonsense you see espoused by so many in the tech and business world today. Both of them are annoying, and I wish they’d stop.

I’m sure there are many arguments to be made against me, but it’s something I feel strongly about. Dryhurst explicitly challenges leftists regarding the adoption of technology without looking to the history of technology and the internet in particular. It was techno-utopian “thinkers” like him who brought us the “decentralized” and “uncensored” internet as it exists today in all its glory. The insistence on technological solutions to political problems, at least from a leftist point of view, is foolhardy.


after a little more consideration i’m a little embarrassed about posting this in the first place. the phrase “AI” seems to be used purely as clickbait here - for me (and probably for many people who will encounter headlines about this project), talking about “AI” takes me out of reality and into a sci-fi daydream almost immediately. it seems like it could be a cool ML tool, but i can’t help but feel the project is selling a lot of… snake oil. or at least it isn’t being honest about what’s really happening and is relying on the audience not thinking about it too much. and it seems to be working! this project seems to have been picked up by virtually every publication that covers music news.

note to self: think for another minute before posting (lol)


No, I’m sorry, because I’m just spamming my opinion here and by no means intended to delegitimatize your post or the work itself. I think the technology (both in the case of blockchain and ML) iis actually really interesting - what I don’t buy is when it’s sold as somehow “revolutionary”, especially by people who profit from it directly.


Autechre on AI:

Sean Booth : I wouldn’t say it’s a living entity, really. It’s about as much like as an entity as a shit AI in a game is. That’s how intelligent it is, which is not intelligent at all, but it might at least resemble the way a person thinks. It’s funny, I’ve been reading about Markov models and Markov chains recently, the results from Markov Chains are remarkably similar to what you get out of Watson or DeepMind, these super advanced language modelling things. And this article was about how unwieldy that kind of mega-gigantic, expensive AI is, because you can actually achieve very close results using Markov chains, and they’re really fucking simple, they’re computationally really easy to deal with, they’re what people use for Twitterbots and things like that. So in some ways these simple conditional responses can resemble very high-end AI. Even though it’s very simple, the result is close enough not to matter.

From this interview.


The best “hard” AI in music I’ve heard is the Yasunao Tone ripper on mego:

Otherwise, I find AI music kind of unremarkable. The Actress/Young Paint thing, for example didn’t do much for me. Haven’t really unpacked why really. I seem to prefer the Autechre style “soft” AI - ‘it’s just a bunch of “if” statements’ type thing


I was excited to see this thread pop up here, but a lot of the commentary has really heavily bummed me out. I get that AI is a bit buzzword bingo, but it’s 2018, not 1988; we should reasonably able to hold a coherent conversation about the field’s algorithms being applied to art without being sexist and dismissive.

And while the “snake oil salesman” comment may match your reality, it’s both aggravating and depressing that even here on lines, a woman’s work is reduced to criticisms about her partner’s.

There is so much more interesting potential to think about and discuss here than one of the artists’ boyfriend, or whether or not AI is an appropriate acronym to use in place of ML.


I agree that a discussion of the work shouldn’t devolve into sexism or unrelated comments about her partner, but I just wanted to add that I saw Herndon and Dryhurst (along with another collaborator whose name I can’t remember right now) discuss this exact work when it was still under wraps at CTM earlier this year and there was no doubt that it was a highly collaborative project. Yes, it’s been released under her name, but it was presented as a work in which the three of them had equal authorship and control.

Now threads like this are (one of many reasons) why Lines is great!

I’ll have to write a proper response to this once I have a moment to sit down, but I’m all in favor or someone like Herndon using her position as the recipient of institutional support to further the conversation re: ML/AI + cultural production in the public sphere.

I mean, I suppose I can appreciate the read of this ‘sensationalizing’ the AI element of this for HH’s own benefit, but I don’t buy that as the primarily motivating factor in the work or its publicization. (also it’s a banger, and jlin is amazing!)

Also, ditto @analogue01that the Yasunao Tone works are incredible (saw a performance of this at Gavin Brown Enterprise and pretty much ripped my face off).

With that being said though, my hunch is that HH’s intent with this work is a bit different, and I appreciate her using her ‘platform’ in an intentional + public way.


One more quick thought:

I think one of the reasons ‘fine art’ in the age of culture-as-content is still important is that it allows us to look into the poetics/problematics of new systems + spaces in a way that fosters critical dialogue and investigation.

This could be why folks like Ferraro or Rafman can contribute meaningfully to this dialogue without actually developing their own AI’s like @ht73 mentioned above (they’re dealing with the ‘affect’ of AI?)

I think HH + MD’s approach to this is different but equally important as members of the institution that also exist as public facing cultural producers rather than straight ahead academics or industry engineers.

Whether that overlap will bring actionable change, i donno, but fingers crossed i guess?

1 Like

I see this slated as a bad joke by industry engineers over in twitter…as for them its clickbait for AI when its actually not, but for the eternal catchphrase of HH/MD and the 3.person nobody ever knows abt (yes, they always act as an entity,for gender conformity?) BUT what is that catchphrase? I still dont really get it. I like Herndons music per se, but i always had difficulties taking in their ‚constructive criticism‘ of the internet. To me this always ended in being outdated by the industry-never being able to actually get a point across before the industry crushes catchphrase technology enabling by non industry ppl…the blockchain/cryptomoney/copyright triangle is a very good example where so much did go wrong in so little time that it’s actually way better to call this a distopian semifuture and abandon the fact that we humans have to fight for integrity within the machine world. And yes this has been done by Ferraro in a very convincing human way. As for the video, i haven’t listened to it just now, i need a human that tells me ‚patrick i know your taste, you‘ll like it‘ because up to this point i see ppl hate or love it in this virtual world but it hasnt sprung to my ears yet…maybe this is HH/MD/3.s agenda?

Check out AIVA (Artificial Intelligence Virtual Artist).

It’s an electronic composer that specialises in Classical and Symphonic music built on deep learning and reinforcement learning architectures.

1 Like

The neural-network chip forms the heart of the synthesizer. It consists of 64 non-linear amplifiers (the electronic neurons on the chip) with 10240 programmable connections. Any input signal can be connected to any neuron, the output of which can be fed back to any input via on-chip or off-chip paths, each with variable connection strength. The same floating-gate devices used in EEPROMs (electrically erasable, programmable, read-only memories) are used in an analog mode of operation to store the strengths of the connections. The synthesizer adds R-C (resistance-capacitance) tank circuits on feedback paths for 16 of the 64 neurons to control the frequencies of oscillation. The R-C circuits produce relaxation oscillations. Interconnecting many relaxation oscillators rapidly produces complex sounds. Global gain and bias signals on the chip control the relative amplitudes of neuron oscillations. Near the onset of oscillation the neurons are sensitive to inherent thermal noise produced by random motions of electron groups moving through the monolithic silicon lattice. This thermal noise adds unpredictability to the synthesizer’s outputs, something David found especially appealing.
The synthesizer’s performance console controls the neural-network chip. R-C circuits, external feedback paths and output channels. The chip itself is not used to its full potential in this first synthesizer. It generates sound and routes signals but the role of learner, pattern-recognizer and responder is played by David, himself a vastly more complex neural network than the chip.



David Tudor rules. idk how I forgot about that one

This is a fun take on using a RNN trained on rap lyrics.


It’s difficult to introduce ‘AI’ into a conversation without people spinning off into philosophical rabbit holes, but if we focus on music/culture, things become a lot simpler. Turing machines are an extremely powerful technology. They have already had a huge impact on music and utilizing more ‘novel’ algorithms(markov chains, neural nets, genetic algos, blah blah blah) is just a continuation of that.

Like any other tool(drum, piano, synthesizer, DAW), it is ultimately the job of the composer to use it creatively to write music. The fact that autechre can produce what they do using mostly if/then conditionals is a testimony the importance of this.

The only paradigm change I think ML brings to the table is that emulation of past works because almost trivial. I have already heard eerily spot-on examples of fake Beatles songs trained on a corpus of Beatles tracks. Humans are already very guilty of this mimicry(see most sub-genres of music), it’s now just becoming more efficient. Perhaps this will push us, as species, to prioritize originality more than we currently do?

Not to beat up on ML, of course. Again, like any technology, there’s nothing stopping people from using it in novel ways. How long before someone trains a convolution networks on a mixture of Taylor Swift and Merzbow?