ReaCoMa - Fluid Corpus Manipulation in REAPER

[jamesbradbury93]

ReaComa

Fluid Corpus Manipulation bindings to the REAPER digital audio workstation.

The last few months I’ve been working on a project ReaCoMa which bring the wonderful FluCoMa tools to REAPER. The tools themselves enable some pretty interesting sonic possibilities that would otherwise be hard to implement or don’t yet exist outside of more research oriented environments such as Python or C++ including:

• Non-negative matrix factorisation (blind source separation)
• Harmonic percussive source separation
• Transient extraction
• Sinusoidal resynthesis
• Novelty slicing
• Two amplitude slicers
• Transient slicing
• Spectral slicing

I’ve also created a series of tutorials that cover installation and usage to help make the process of integrating ReaCoMa into your creative workflow much easier. The playlist of these can be found below. I plan to develop on these tutorials in the coming weeks to go into more detail for each of the algorithms and to talk about how you more developer-y types can extend what is already there.

Of course all of the software and tools are open source and free for you to use as you please, licensed under BSD3. More specific details and links can be found at the home website (unfortunately I can only do 2 links in this post as I’m fairly green!).

Requirements

REAPER
FluCoMa command line tools

Documentation

Step 1: Command Line Tools

All of the number crunching and DSP is performed by the FluCoMa command line tools. You will need to download the appropriate version for your operating system. Precompiled binaries can be found here or you can compile them from source.

Once you have a set of the binaries you will need to store them in a sensible location that is likely not to change.

Step 2: Lua Scripts

The lua scripts orchestrate calls between REAPER and the command line executables. They take care of things like providing a basic user interface and the specification of parameters for each algorithm.

1. Download all of the files from this repository.

https://github.com/jamesb93/ReaCoMa/archive/master.zip

2. Unzip the downloaded archive and you will have a folder contaning a number of files ending with the .lua extension - these are the scripts that relate to each FluCoMa algorithm.

You can have this folder anywhere you want on your machine, REAPER doesn’t actually care too much about where a script is launched from and I have made sure that any dependencies are sorted out in-house and not as part of your REAPER installation. The entire thing is very portable.

When you try to run the lua scripts for the first time it will ask you to provide the path where you stored the command line tools. This creates a semi-permanent link between the scripts and ReaCoMa using internal extended states in the REAPER ReaScript API.

Download

https://www.reacoma.xyz
Source Code @ GitHub

[tremblap]

I won’t do a fake flucoma.org account to tell you how much I love this. I use it all the time (even if I have all the cool toys in Max SC and Pd) because it is so immediate! @Rodrigo likes them too I think?

[Rodrigo]

I’ve been playing with these a bunch as they’ve been in development. Suuuper handy.

Makes me wish I had more use for Reaper…
Sadly(/thankfully) all I really do with a DAW these days is use it as a dumb “tape machine”, so much of the super detailed and fancy-pants stuff you can do in Reaper is wasted on me.

[_mark]

Wow - some really powerful stuff there. I’m definitely going to play around with this. Have been looking for more reasons to learn Reaper too…

Thanks for making it available.

[jamesbradbury93]

If you do come across any bugs let me know via the github issues or informally here. I’m also really interested if you hit a barrier somewhere and want to use ReaCoMa to push further in a particular direction - that gives me some fun coding work to do

[_mark]

Will do.

I’m really interested in it for more experimental processing, but I’m also curious whether you think there is potential to apply the tools to post-production style work, eg noise reduction?

[jamesbradbury93]

Totally!

I use it all the time to ‘clean’ sounds to a point where I get what I want out of them. In particular you might find the fluid-transient and fluid-nmf algorithms immediately useful for extracting noise/transients/clicks from sounds.

fluid-transient uses a regressor to determine when an outlier in the signal occurs with varying ‘sizes’ of the analysis and different levels of sensitivity.

fluid-nmf is a canonical approach to source separation, but you could also use it for finding the noise and trashing it. I’ve had good success with fluid-hpss which is a harmonic percussive separation algorithm. If what you want to retain is mostly harmonic this can be a nice way to clean it. Of course you can also chain these processes together by running the algos in different orders. You might want to do something like hpss > transient > nmf into some gating.

For other kinds of post production work I would highly recommend you get into creative experimentation with nmf. The separation allows you to split up your audio and process parallel streams in so many interesting ways. Same goes for transients too.

[_mark]

Thanks for taking the time to write those up - that all looks useful and interesting.

Unfortunately though, I’m getting application not responding every time I execute some of the scripts.

Following your video here: https://www.youtube.com/watch?v=HcF_lssW2Tg&list=PLCQRw62RgghbsZgsA98lLkOwjBSs4yc9T&index=2

All good up until 5.21 - then I press ok and the application locks up. I’ve tried nmf, hpss and sines with the same result. Oddly ampgate and ampslice seem to complete, but I can’t see it has done anything to the audio. noveltyslice seems to work though - I can see some slicing has occurred!

I’m on:
Reaper 6.11/64
OSX 10.14.6

I’m interestingly seeing files appear in the folder of the source file though:

Happy to provide more info if helpful.

[jamesbradbury93]

I think this is an easy one, very kind of you to let me warm up first

Are these super long files? Assuming you are running defaults (2 components?) then these are one minute sound files (10mb per min * 2channels). If you can open these sound files by dropping them back into REAPER then its likely the process is just taking a long time and not finishing in a reasonable timeframe for you. NMF is generally a super chunky algorithm to run and its complexity spirals with the fft size and the number of iterations.

Can you try this for me:

1. Small sound file 2-10 seconds
2. Iterations 50
3. fftsettings 1024 512 1024

and see if it runs without locking up forever?

If you are able to send me the source media that you are testing and the parameters I’ll be able to rule out a machine specific error too but I think its more likely that you are just asking the algos to work through a lot of numbers.

[_mark]

Thanks for the reply. Yes, you were right. That file is 15 seconds (but also 192khz), and takes around 1min 30seconds to process with default nmf settings - so more patience required my end!

In case it’s still helpful, here is the file:
https://drive.google.com/file/d/1ImbriGO-4P0yMqBS1csXH1-5aUc6QHpf/view?usp=sharing

[jamesbradbury93]

Ah yes, high sample rate too will make the computation that much more complex! I tend to experiment with very low number of iterations and a cheap-ish FFT then when I think I’ve got something good I’ll go for a mega-pass with chunky settings and go make a tea. You might also let REAPER guide your intuitions and then run the command line stuff elsewhere so that it doesn’t block the GUI and you can keep working. Perhaps there is room here for me to make a process non-blocking with an option flag or something. The only problem is if you move the item inbetween processing starting and stopping things could go wonky.

[wheelersounds]

Hi @jamesbradbury93, I’ve got the ReaCoMa scripts pointing to the binaries but when I try to run them, I get the following error:

Summary

..._15950227959741.wav failed to be made by the command line.
stack traceback:
...tion Support/REAPER/Scripts/ReaCoMa-master/lib/utils.lua:103: in function 'utils.assert'
...ion Support/REAPER/Scripts/ReaCoMa-master/lib/layers.lua:29: in function 'layers.exist'
...tion Support/REAPER/Scripts/ReaCoMa-master/fluid-nmf.lua:57: in main chunk

I’m wondering if I did something wrong in setup? Thanks! Really excited to use these tools.
EDIT: Apparently I need to convert files to WAV or AIFF before running the scripts – the first file I tried was an .MP3, but I just tried on a WAV and it worked like a charm. Thanks!

[jamesbradbury93]

Ah yes, I should document this better to save debugging time in the future. Thanks for bringing that to my attention. I’m glad you persisted and got the scripts to work! Let me know if you have any questions.

[trickyflemming]

Thank you so much for putting this together! This is great. It reminds me a lot of the Composers Desktop Project wrapper for Renoise in that it takes a really cool command line tool and gives it a useful, integrated interface in the DAW. I’ll be using this a lot.

[Larry]

This is so good! I got it working without a problem today (exept for trying to run the scripts on a .flac first) and had so much fun exploring the nmf script with different material. This’ll keep me busy for a while I’m sure.
Thanks a lot for this!

[jamesbradbury93]

I think in the next release I’ll try and make a prompt if the source is the wrong format. It’s really useful that you guys have mentioned this in your experience because to me I’m so used to running the tools on wav files that I don’t catch those sticky bits of user experience.

[tremblap]

@_mark

I’m also curious whether you think there is potential to apply the tools to post-production style work, eg noise reduction?

actually, my obsession with null summing allows you to demix and do either subtle changes almost imperceptible… or some seriously nasty destructive processing too Sharpening or smoothing by a few dB tend to be a very potent example of the former (fft artefacts are quite masked with subtle stuff) and processing of certain elements only (demixing for processing) is a good example of the latter.

[_mark]

So would that be using the nmf algo? You are saying to de-mix with this, then working on individual elements before mixing together again?

I’ve had some fun on more experimental stuff, but need to try with some voice.

[jamesbradbury93]

Just to chime in here:

All of the algorithms null-sum so whatever you extract/separate can be added together to recreate the original. So you might NMF to 2 components (in the hope that one of the components captures the noise) and then attenuate that component and add back to the other component to get a less noisy version of the original.

[_mark]

I see what you mean. I’d wondered if that was the case, so thanks for confirming.

Another question along those lines - with nmf, does it define a component of the sound based on analysis of the whole file or is it more step by step? So in practice, will I get different results working on 10 second chunks instead of processing a 60 second file for instance?