#1

Using Docker to run Norns

See this GitHub repo:

If you have Docker running on your computer (tested with Linux and OS X), then you can run this command:

docker run --rm -it \
    --ulimit rtprio=95 --ulimit memlock=-1 --shm-size=256m \
    -p 5000:5000 -p 5555:5555 -p 5556:5556 \
    samdoshi/norns-test-dummy

(FYI: it will download 500MB-1GB of data)

and visit http://127.0.0.1:5000/maiden/ in your browser. There is no audio.

Type ctrl-b d to exit.

For working audio and grid on Linux see norns-test-dummy.

Audio on OS X

Iā€™ve had a brief experimentation trying to run NetJack2 on OSX, but I couldnā€™t get the client inside the Docker container to see the host. According to this it works in principle. The issue might be due to how networking works with Docker for Mac.

An alternative idea might be to run scserver and crone outside the container, but run matron and maiden inside. In particular matron is the one that is heavily dependant on a Linux base.

(A futher note, SuperCollider wonā€™t build with the default 2gb RAM that Docker for Mac allocates to the VM it uses, I had success with 4gb.)

Data dump

Audio experiments with containers

See this gist for an example using ALSA and SuperCollider (Linux only).

See this gist for an example using NetJack2 and SuperCollider (Linux only so far).

Useful jack commands

Iā€™m using jack2-dbus on my host computer ā€“ Arch Linux FYI.

Iā€™m also using catia to manage connections, once Iā€™ve got some name stability with the containers Iā€™ll try to automate connections with jack_connect.

jack_control stop    # stop jack
jack_control start   # start jack
jack_load netmanager # load the NetJack2 plugin

# or combined (jack can get itself in a funk and sometimes needs a kick)
jack_control stop && jack_control start && jack_load netmanager

jack_lsp    # list ports (i.e. inputs, outputs and apps)
jack_lsp -c # list ports and connections

Using names with ALSA

ALSA hardware numbers can (and do) change, you can use hardware names instead of numbers when starting jackd.

$ cat /proc/asound/cards
 0 [Crimson        ]: USB-Audio - SPL Crimson
                      Ploytec GmbH SPL Crimson at usb-0000:00:14.0-1, high speed
 1 [PCH            ]: HDA-Intel - HDA Intel PCH
                      HDA Intel PCH at 0x13ffff10000 irq 47
 2 [HDMI           ]: HDA-Intel - HDA ATI HDMI
                      HDA ATI HDMI at 0xfbd60000 irq 51
 3 [PC4            ]: USB-Audio - Faderfox PC4
                      Faderfox Faderfox PC4 at usb-0000:00:14.0-3.2, full speed

Using the above info, you can change a .jackdrc from:

/usr/bin/jackd -R -d alsa -d hw:0

to:

/usr/bin/jackd -R -d alsa -d hw:Crimson

X11 forwarding

How to show X11 windows with Docker on Mac

Debugging inside a container

To run gdb against a process inside a container you must add --cap-add=SYS_PTRACE --security-opt seccomp=unconfined to your docker run command.

As an example, here is how to debug matron, if we wish to debug an interaction with maiden we need to attach to a running process due to using ws-wrapper.

$ docker exec --user=root -it <container name> bash  # get a privileged user inside a running container
# fyi, the following commands are now running in the container
$ cd norns
$ ps ax | grep matron  # find the matron process (choose the one without ws-wrapper)
$ gdb ./build/matron/matron <ps number>
(gdb) continue  # gdb will pause the process when it starts, so type 'continue' to start it again

Useful links

  • This GitHub repo has SuperCollider running inside a docker container, albeit with the sound card passed directly.

Changelog

  • 2018-09-08: Added link to the norns-dev GitHub repo, shortened old sections
  • 2018-09-03: Added some OS X notes, X11 notes, and gdb notes
  • 2018-09-01: Created this first post
2 Likes
Norns technical discussions on lines (& moderation)
#2

What?

I want a reproducible dev environment.

I want to be able to type make run and have everything taken care of for me.

I want to run make clean && make build and have everything new and fresh.

Itā€™s just how I am.

The point of this thread is for me to explore how to make that happen by trying to use docker to do it.

So why a container?

The Norns software assumes itā€™s running on a Rasbian based system with extras. The software is coupled to both itā€™s hardware, but also the base Linux system itā€™s running on.

I want to:

  • Get Norns running with as few changes as possible in an environment as close to the shipped product as possible
  • Insulate the host OS from any accidental damage (e.g. through use of sudo from a Lua script)

Why docker?

Itā€™s common, and the Dockerfiles are easily understood even by users unfamiliar with docker. Itā€™s as easy to use as one can expect given what it does. Also (and this is the main reason) itā€™s the only container software I know how to useā€¦

And maybe via docker-machine Windows and OSX users might be able to join in (albeit within a VM).
(edit: or rather Docker for Mac / Docker for Windows, thanks @jlmitch5)

Why not a VM?

jack gets sad without realtime, if we can avoid the overhead of the VM, that will be one way to make jack happier.

Also, IMO the docker tooling and file format is better than vagrant.

Finally, using docker opens up using it for CI with Travis CI

Where docker may be more of a pain is when using it for iterative development, as itā€™s not really what itā€™s designed for. Iā€™ve got a few ideas for how to reduce the issues, possibly involving splitting up each service up into itā€™s own container.

FYI there is no systemd inside a docker container.

What this thread is and isnā€™t!

Itā€™s Linux first and foremost. Sorry others!

Itā€™s not about alt-devices, mainly itā€™s about getting a reproducible dev environment running.

Itā€™s probably not going to be a short thread, really itā€™s going to be a (my) work log. I will try and keep the first post up to date though.

I work very slow. Anyone that goes faster than me might be asked to slow down, or worseā€¦ Iā€™ll ask you to take over!

I am aiming for an end product. Hopefully a GitHub repo with a nice detailed README.md that others can use to get set up. But there is a very real possibility that it doesnā€™t work (or even falls at the first hurdle).

Also, NetJack2 uses master/slave terminology, sorry about that!

Where am I at?

I have managed to run SuperCollider inside a container, and have forwarded the audio via NetJack2 from the container to the host. Iā€™m going to try to get the code up in a gist in the next few days (I would like to get rid of the hardcoded IP addresses first).

Whatā€™s next

Post my proof of concept for running SuperCollider in a container.

Try and get the norns software installed and running inside the container.

Sticking points

How to deal with the screen and buttons? I have ideas, but this is the one area where changes will be needed in the software.

Deal with connecting peripherals!

Data dump

Useful jack commands

Iā€™m using jack2-dbus on my host computer ā€“ Arch Linux FYI.

Iā€™m also using catia to manage connections, once Iā€™ve got some name stability with the containers Iā€™ll try to automate connections with jack_connect.

jack_control stop    # stop jack
jack_control start   # start jack
jack_load netmanager # load the NetJack2 plugin

# or combined (jack can get itself in a funk and sometimes needs a kick)
jack_control stop && jack_control start && jack_load netmanager

jack_lsp    # list ports (i.e. inputs, outputs and apps)
jack_lsp -c # list ports and connections

Useful links

  • This GitHub repo has SuperCollider running inside a docker container, albeit with the sound card passed directly.
10 Likes
#3

Just a small note that this has been replaced by ā€œDocker for Macā€ and ā€œDocker for Windowsā€ (and the Mac one at least is a lot more stable and performant then things used to be).

1 Like
#4

With some help from @rbxbx last night I started trying to get a norns install working with Vagrant / Virtualbox. I figure this will have similar issues so I want to keep an eye on this thread. :smile: :

2 Likes
#5

Iā€™d love to know how the audio performance is once you get to the point of being able to tell (even if itā€™s just running SuperCollider).

I know one hack is to try and pass-through a USB sound card. The NetJack2 stuff may also get you over the line of ā€˜good enoughā€™.

There will definitely be a lot of overlap, Iā€™m still up in the air over whether docker or vagrant will have better ergonomics, but I figure once we have one working, it will be relatively trivial to switch to the other (performance issues aside).

1 Like
#6

Got stalled on vagrant install with debian/contrib-stretch64.

Norns ./waf configure barfs with an error it canā€™t find nanomsg.

Guessing that norns-image has arch-specific packages for arm that wonā€™t work in vagrant (as vagrant doesnā€™t support arm)?

#7

Some folks have set up dev environments that will cross-compile to an rpi arm target:

But thatā€™s a different ball of wax from running an rpi image in a vagrant box, directly.

Maybe a new x86 arch norns image will be necessary for this effort? (is that exactly what @sam has in mind?)

#8

this can all be built from source , its detailed here:

as mentioned elsewhere, I have crone/matron/maiden (so full norns) all running on a VMware Linux VM on my mac, done in precisely this way.

the only thing thats not really mentioned in the above doc is you need SC 3.9.0 (and sc3-plugins?) , and whilst this has been released, its not on all distros yet - so you may have to build that too.
(but thats simple, its detail on the SC GitHub page, and you can just take the defaults)

after than you just have to set .jackdrc correctly

you can run from scripts, thats no issue - systemd is only a convenient way to auto start things.

lol thats where the fun starts :wink:

2 Likes
#9

So the version of SC is installed by apt-get is not gonna work?

sudo apt install supercollider-language supercollider-server supercollider-dev    

1:3.7.0~repack-4

I assume thatā€™s 3.7.0?

#10

Going to chime in with a link to Packer, which is a way to describe/document reproducible builds regardless of virtualization/containerization tech (supports Virtualbox, Vagrant, Docker, etc., plus cloud platforms)

This plugin might be interesting ā€“ being able to run/test a RPi image on desktop and then transfer that exact image to the hardware could be valuable. No idea if Jack/audio support is feasible, however.

4 Likes
#11

The baby was sick overnight, so apologies if my replies are incoherent.

I had to build SuperCollider from source to get it to run inside Docker, else it complained about a lack of X11 (need to disable QT), the following worked for me (pasted directly from the Dockerfile):

ENV SC_VERSION=3.9.3

RUN mkdir -p /tmp/sc && \
    cd /tmp/sc && \
    wget -q https://github.com/supercollider/supercollider/releases/download/Version-$SC_VERSION/SuperCollider-$SC_VERSION-Source-linux.tar.bz2 -O sc.tar.bz2 && \
    tar xvf sc.tar.bz2

RUN cd /tmp/sc/SuperCollider-Source && \
    mkdir -p build && \
    cd build && \
    cmake -DCMAKE_BUILD_TYPE="Release" \
          -DBUILD_TESTING=OFF \
          -DSUPERNOVA=OFF \
          -DNATIVE=OFF \
          -DSC_WII=OFF \
          -DSC_QT=OFF \
          -DSC_ED=OFF \
          -DSC_EL=OFF \
          -DSC_VIM=OFF \
          .. && \
    make -j && \
    make install

Most of that is more or less copied from the GitHub repo listed at the bottom of the second post in this topic.

Thanks for that, I think I might have known about it, I thinkā€¦

I suspect my short term memory wonā€™t be recording today, so can someone remind me of itā€™s existence when the time comes to package our investigations up into a useful product.

Now that is pretty cool. Iā€™ve always been really frustrated by the lack of sensible tooling for building Raspberry Pi images, most of it seems to be ā€œdo it on the device, then image the SD cardā€.

It will be interesting to see where this goes. Convergence between development environments and final image building would be awesome.

Just summerising for myself, there is the potential for:

  • Docker dev env using NetJack2
  • Vagrant x64 VM
    • using NetJack2
    • using USB passthrough
    • using emulated audio
  • Vagrant ARM VM
    • using NetJack2 (crossplatform NetJack2?)
    • using USB passthrough
    • using emulated audio
  • ARM image building

For now, Iā€™ll stick with the first, we donā€™t want to get too ahead of ourselves!

1 Like
#12

FWIW - (after compiling sc 3.9.3) I got norns running (kinda) tonight and got audio from the linux box to come out of my mac speakers, but ran into a few issues with the various norns services not getting started on boot. Probably will start with a fresh vagrant box install tomorrow to be sure I have all the steps documented.

Noticed this error on running slcang or crone

sc3> ERROR: API version mismatch: /home/vagrant/.local/share/SuperCollider/Extensions/norns/BufWrPre.so
    This plugin is not compatible with SuperCollider >=3.9.0
    The plugin has not been loaded; please find or compile a newer version.
    (Plugin's API version: 2. Expected: 3)
2 Likes
#13

Probably best to make a ticket for that issue on the Norns bug tracker. There has been much talk about info like that getting lost.

The source for the UGen is here.

Looking forward to seeing the Vagrantfile when youā€™re ready!

#14

Youā€™ve always been able to cross-compile images, itā€™s just a bit of a pain to setup. So often laziness rather than tooling.

But these days thereā€™s easier solutions, as many things like buildroot support it out of the box. I particularly like Armbian, higher level than some, but still a complete environment you just install and run.

Iā€™ve actually been looking at armbian for the Organelle, so that C&G can just ā€˜click a buttonā€™ to build a completely fresh os image - as they created the original by hand then seem to have ā€˜lostā€™ how it was created :wink:

(It would be pretty simple to do for Norns too)

#15

What jackdrc settings are working for you?

I finally got norns making sound, but itā€™s all crunchy/distorted.

#16

Is vagrant the way to go on macs instead of docker? I donā€™t think it can access the native soundcardsā€¦

#17

Sorry for the delays folks. The babyā€™s sick turned out to be a stomach bug which spread around the houseā€¦

The summer holidays have just started in the UK too, so no idea when or what time I will have for a few weeks.

No you canā€™t access native sound cards from Docker for Mac as far as I know. You can do it on Linux, but I believe it means exclusively using that soundcard inside the container.

Itā€™s possible that the same Jack network code that Iā€™m using with Docker might also be able to send audio from the Docker for Mac1 guest to the OSX host and that it might have better audio performance than an emulated soundcard.

I think I mentioned it earlier, but we should be easy enough to translate a working Dockerfile into a Vagrantfile (and vice-versa) once we know what weā€™re doing. So any knowledge towards that goal is useful.

1 Also the same technique could be used with Vagrant

#18

FWIW - I have built norns and got sound working in Vagrant with Virtualbox on MacOS (with Mac native/built-in audio), but Iā€™ve not created a Vagrantfile as thatā€™s a little outside of my skill set so far.

However, the sound is glitchy as hell and pretty much unusable so I kinda gave up on this.

EDIT: I can post my install notes if someone else wants to make a Vagrantfile

2 Likes
#19

I will try to make a working Dockerfile for norns and try this audio approach. Will report once Iā€™ve gotten somewhere :slight_smile:

2 Likes
#20

Hereā€™s a Dockerfile where things appear to compile but portaudio is not setup yet. You can ssh interactively and start crone.sh but complains about audio as well as other stuff. May be easier to use NetJack2 for the time being but hopefully this is a good start:

1 Like