Synthetic Speech and Vocals

Hi all,

I just stumbled across this wonderful video about Animal Crossing’s psuedo-speech that offers a nice survey over other methods for masquerading text+sound as speech.

The video makes an interesting point – when we see text appear chunk-by-chunk accompanied by sounds, most folks will be able to follow along easier than if presented with the text all at once.

It got me thinking – does anyone here have any experience with any of these techniques? Would people be willing to donate small snippets of speech so we can have a library to provide voices to text?

16 Likes

Interesting!
You may find this relevant

There was a synthetic speech element in a recent
STBB Forever beat battle.

I posted my entry here:

I used ttsmp3.com
It was great techniques for make speech, I will use again.

There are other online speech generators here:
https://ttsmp3.com/


http://www.fromtexttospeech.com/
http://onlinetonegenerator.com/voice-generator.html

To see other things make by other great people in this
battle here is full post:
https://www.stbbforever.com/viewtopic.php?f=2&t=568

3 Likes

Wowowow digging Mach a lot!

Thanks for the resources <3

1 Like

I’m a big fan of speech synthesis, and I love to make things babble! I also love to babble about things babbling. There is something I find so charming about getting a computer to chatter. One of my pipe dreams is to build an asemic speech engine that is built around prosody and inflection.

I once wrote a study called “computer on the phone with his mother”:

I wrote this in my Sporth language. You can find the sporth code here.

The recipe to make speech sounds like this is reasonably straight forward to replicate in a modular synthesis environment (Csound, SC, PD, FAUST, etc). What this is doing is taking a narrow pulse wave and putting it through a series of bandpass filters tuned to formant frequencies. Interpolating between the formant values gives it the “talky” effect. Randomly interpolating between formant values makes it “babble”. I also added some pauses, which makes it feel more like words in a language. The breaks are what give it personality (similarly, in singing synthesis, choosing the right vibrato makes a world of difference).

About a year or so later, I made another babble track. This one uses a Kelly-Lochbaum vocal track to synthesize the voice. From a control standpoint, the general concept is very similar, only instead of moving a bunch of bandpass filters around, you’re pushing and pulling on a virtual vocal tract to shape the sounds. The implementation itself is based on pink trombone, but I hastily ported the javascript code to C to make it work with the rest of my music software ecosystem.

There’s also stuff like espeak and festival which I’ve used in the past to generate electronic speech.

22 Likes

My partner and I built Martin Howse’s ‘Wormed Voice’, seems promising and the documentation is a treasure trove of speech synthesis history + info http://1010.co.uk/org/wormedvoice.html

Hers works, mine doesn’t properly :-/ gonna try and build another and see what went wrong

I also got a Micro:bit a while ago to experiment with it’s very limited speech synth possibilites, did a couple of little tests but yet to properly get into it.

Years ago I had a dream that I found a rackmount hardware speech synth machine at a car boot sale, recorded a 7" using it and John Peel played it… :scream:

10 Likes

Thanks for all of this. I’m excited to see what others have been using as well. I’ve been experimenting with synthetic speech as well and a lot of these notes are inspiring. One point of inspiration has been the group Visible Cloaks and their use of synthetic speech that generates MIDI data, which they refer to as ‘imprinting.’ Here’s a recent interview that they discuss the process:

Ryan brought up the idea of using the “text-to-music” (TTM) function in the generative music program Wotja to create the starting point for a piece. It’s an extension of a creative process I have come to call “imprinting” — using text or musical fragments to generate a digital representation of it in MIDI, then using it as a building block in the construction of a more complex arrangement.

Link

5 Likes

You can also get pretty far by repeating spoken word over and over again. Your brain will eventually turn it into music. Diana Deutsch has a really great example on her website.

Steve Reich employs speech-to-note transcription in Different Trains, which I’ve always enjoyed listening to (especially on trains or any moving vehicle really).

Tangentially related, but here is an entertaining video of a young Sir Ian McKellen performing different interpretations of the opening of “the merchant of venice”. I’ve always wanted a music sequencer that could behave like this.

7 Likes

…bringing to mind some potential Norns script ideas…

3 Likes



7 Likes

Thank you!
I’m glad you have enjoy!

Also, am enjoy this thread, myself, thanks for starting!

1 Like

Great things Paul!

big favorite is pink trombone
I sample this often in my digitakt for make weird wobbly
synths.
I love play with it when listen to music also, is like the uncanny valley
of robot vox! I imagine a poorly render robot comes to town singing
all your favorite medieval hits!
aaaaaaaw waaaw wawawa lalaaa llaa la!

2 Likes

So if I were to make a series of samples for these purposes, what would people like? Syllables? Rare phonemes?

1 Like

There is a very cool device that is exploring those concepts: Pocket Operator PO-35 speak. It’s maybe overly minimalistic to make entire tunes in but as an immediate voice sampler with vocoding, vowel synthesis, and autotune it’s rather impressive.

Playing with it enables you to employ some of the creative strategies that @PaulBatchelor talked about above. If you’d like I can record a demo.

There’s a pretty good interview with the creator of the device that includes some technical detail:

1 Like

There is some similar discussion on the topic in this thread

1 Like

In case you want to go animal crossing style (in the browser!) :smiley: https://acedio.github.io/animalese.js/

2 Likes

Bitspeek and the PO-35 use Linear Predictive Coding, is another fun way to do computer speech.

Some time ago, I found an open source implementation of the LPC10 codec, which I believe is very similar to the LPC ICs found on the speak-n-spell.

I built a little filter around this in soundpipe that takes in a streaming input signal (assumed to be speech), and converts it to a that computer sound via openlpc.

Here’s what it sounds like (original plays first, followed by the LPC-processed output):

5 Likes

I was obsessed with LPC with RTCmix in the 1990s and then i had some bad-ass LPC externals for MAx which i think are only 32bit

It’s such a nerdy sound i cant resist it, i went down a huge rabbit hole with bytebeat too which is in the same universe for me sonically …lol

2 Likes

This is the one
Paul Lansky – of RadioHead Fame was our Composer in Residence and he showed us this
piece and he played to a group of grads and undergrads the first .mp3 of a file
and then he sent it to everyone in a mail and we all were floored …hahahaha

1 Like

I just wanted to say thank you for the Diana Deutsch link - very interesting to me! I’ve been experimenting with singing poetry in the past year or so, and this experiment really resonates - it seems a very clear example of how deeply linked song & speech seem to be.

I experienced the effect of the non-repeated phrase seeming to “sing out” in the second example - but somehow I can’t think of the effect as necessarily illusory - it feels to me more like an exaggeration of something that is already there, that the initial repetition sensitises one to the musical/tonal quality already inherent in the speech, only occluded by our habit of just focusing on the words, as it were… maybe it’s our usual way of hearing speech that’s the illusion. :upside_down_face:

(It reminded me also of an experiment/workshop-thing I have a recording of John C Lilly doing, which is a single word (“cogitate” in this instance) looped over & over - doesn’t take long before most people start hearing a whole range of different words/short phrases.)

(& random things about whistled/tonal languages, mythical stories about the secret wisdoms of birds, and contented gorillas humming happy songs to themselves as they eat, eatc)

2 Likes

what do you think about Arnold Schoenberg’s “Pierrot Lunaire” and sprechgesang? i think that’s the term i always found it poetic and very compelling