Norns Midi clock to ALL connected devices(solved)

[tekknovator]

Hi, it seems that I have to decide for a single port to send clock in “PARAMETERS > CLOCK > midi out” menu.
Is there a script independent way to send the norns midi clock to all configured midi device ports?

Here is what I try to setup:
I have a very powerful USB Hub (Amazon Basics 20v, strongest I could find) where all my midi devices are connected, including the Norns (for power).
Norns is also connected as the host of the hub. All connected devices show up as expected. Now I want to sync them all, but norns menu does not offer a broadcast.

So I turn to you great people for ideas

[zebra]

the feature would have to be added in norns system code. fortunately it is a simple change.

are you interested implementing the change yourself, or just testing?

[tekknovator]

@zebra I wouldn’t know where to start, but if it is a simple change I might try to get my feet wet and give it a go. Is there some developer onboarding doc anywhere?

[zebra]

hrm hmm. well there are the scripting docs, which come in two flavors (workshop-like, and reference-with-examples.) but these are focused on writing scripts / consuming the system API, not developing the system.

tbh i do not know the docs super well! and am not sure if we have something like a developer setup guide; if not we should make one? (how to ssh, run/restart stack, lua syntax cheatsheet?)

also the “norns study group” discord server is a good place to ask realtime questions if you are into that platform.

well in any case, the changes would be: (this stuff is found under ~/norns/lua/.. on your device):

• add a “broadcast” or whatever option to this variable called clock_table, after it is filled with midi ports.
  [ norns/clock.lua at main · monome/norns · GitHub]
  (and hey while you’re at it give it a better name like clock_midi_out_table)

• in the midi out update function, test for that option and perform :clock() on every connected device:
  [ norns/clock.lua at main · monome/norns · GitHub ]
  (while you’re at it you could refactor that guy for performance as the FIXME suggests! if you’re feeling ambitious.)
  (note that this will only send to devices that you have selected in the midi ports menu, which i’d call a “feature”)

… and i kinda think that’s it? state persistence should continue to work. i don’t even think it would affect existing presets.

oh, these changes will take effect on restarting the matron process or rebooting norns.

of course i can also just make these changes the next time i get a few minutes with the norns, it would be blind since i don’t happen to have appropriate hardware to test. so if it sounds fun then go for it by all means.

or of course someone else may come across this post in the meantime.

[Dewb]

I think you probably could implement this behavior in the lua side if you were willing to modify each script you wanted to use; just insert something like this in a clock coroutine that’s running at 24x quarter note speed.

    for id, device in pairs(midi.devices) do
        device:send({248})
    end

Adding all-device clock broadcast support to the core clock has been on my todo list because it was a critical feature of the older beatclock library for me, but I’ve gotten distracted by other projects lately.

Edit: I think this discussion has given me sufficient brainworms to take a swing at a PR for this tomorrow if nobody else gets to it first.

Edit 2: An important wrinkle here is that if norns is set to receive clock from an external MIDI device, you don’t want to echo clock back to that device.

[tekknovator]

Nice, might give it an hour or two today. Thanks for the rundown @zebra, much appreciated

Edit1: Found this, it has some links for build setup etc. I am going to explore now.

GitHub - monome/norns: norns is many sound instruments.

Edit2: Exploring the environment.

• Connected to norns via ssh and changed default password.

• vim is available

• trying to connect vscode(in vim mode of course ) works flawless too(as expected).
  

  

• installed prerequisites via apt-get install on the pi

• trying to build Nanomsg: what? no cmake?

• installed cmake on the pi

• built libmonome

• built Nanomsg (after diving down the Nanomsg/NNG rabbithole, quite inspiring lib indeed)

• building norns (sidenote, the waf scripts are really comfortable. <3 )

Stupid me, trying to compile directly on the norns takes ages

Edit3: ‘build’ finished successfully (1h56m42.405s), that was quite a while. Poor pi produced quite some heat.

[tekknovator]

This is madness, but I love it. Do I get it about right?
(sorry for dabbling in drawio, only saw later that graphviz is used in the project)

norns_architecture.drawio (11.7 KB)

[tekknovator]

@zebra I implemented like you suggested, it and it works great (tested with circuit and lsdjmc as followers). Thanks again for your help.

@Dewb Unfortunately I solved it by stuffing even more logic into the function. How did you measure the time that made you write the fixme comment?

Here is what I did hack together after work today:

Edit: Some thoughts and questions regarding the performance:

• I suppose integer comparison is faster then string comparison. Or is it all just table comparison from a c perspective?
• does joining lines 278 from the screenshot and line 279 into a single AND condition gain any speed?
• can we avoid iterating over all vports?

Idea: filter the midi_out table for vports with a name other then “none” into a new table, outside the update function, and iterate over that one on broadcast. It will still not deal with the name check and the broadcast decision. Sorry for sounding convoluted I will just hack and measure, once I know how.

[Dewb]

I didn’t write that comment, but the way I would approach this is to do as much work as possible in the parameter update function. Things that only change there can be tested there. You have to keep the condition that checks whether there’s a device on a port, because that could change outside the parameter update function, but the list of which ports you want to send clock to is knowable in advance. So I’d maybe create a table of devices that should receive clock, populate it whenever the clock source or destination changes. If the clock source is a midi device, make sure it gets removed from the list. Then in the 24ppq clock routine, just iterate over that list (which will have 0 or 1 elements unless you’ve chosen the “all” option.) There are probably additional optimizations you could make to speed up the 0 and 1 case (like stopping the coroutine entirely when no devices need clock, or substituting the clock function with a different function that doesn’t iterate in the 1 case.)

[tekknovator]

Nice ideas, hope to find some time to think about that tomorrow. Now I make some noise

[zebra]

@tekknovator

Edit3: ‘build’ finished successfully (1h56m42.405s), that was quite a while. Poor pi produced quite some heat.

whaaaaat that’s not at all expected if you’re just compiling the norns stack. we always build on device and it usually takes, i dunno, <10min (<5min?) for a clean rebuild.

(of course make sure you stop the norns stack running first! also ./waf -j 3 is useful.)

BTW… you don’t have to recompile anything if you’re just changing lua code.

[zebra]

pretty close. i would draw it something like this.

(quick ugly informal graphviz version, hopefully it makes sense)

code for graph

digraph G { 
  rankdir=TB
  node [shape=box];
  edge [arrowhead="none"];

    
    subgraph cluster_0 { 
		style=filled;
		color=peachpuff;
        matron;
        node [shape=component] weaver;
        node [shape=component] lua;
        matron -> weaver [style="dotted"];
        weaver -> lua [style="dotted"];
        label="matron process";
    }

    subgraph cluster_1 { 
		  style=filled;
		  color=lightblue;
        softcut -> mixer[style=bold];
        node [shape=component] osc_server[label="OSC server"];
        osc_server -> mixer [style="dotted"];
        osc_server -> softcut [style="dotted"];
        label = "crone process";
    }

    scsynth -> mixer[style=bold];
    
    sclang -> scsynth [style="dashed"];

    maiden;

    node [shape=Mdiamond] ADC;
    node [shape=Msquare] DAC;

    ADC -> mixer[style=bold, arrowhead=normal];
    mixer -> DAC[style=bold, arrowhead=normal];

    matron -> maiden;
    sclang -> maiden;

    matron -> sclang [style=dashed]
    matron -> osc_server [style=dashed]

    node[shape=ellipse] user;

    node [shape=folder, label =".sc"] sc_files;
    node [shape=folder, label = ".lua"] script_files;

    maiden -> script_files;

    user -> sc_files -> sclang;
    user -> script_files -> lua;
    user -> maiden
}

code for legend

digraph G { 
    subgraph cluster_2 { 
        label = "legend";
        node [style=invisible];
        sink1;
        sink2;
        sink3;
        sink4;
        node [shape=plaintext, style=solid];
        JACK;
        OSC;
        websocket[label="websocket/etc"];
        subcomponent;
        
        JACK -> sink1 [style=bold];
        OSC -> sink2 [style=dashed];
        websocket -> sink3 [style=solid];
        subcomponent -> sink4 [style=dotted];
    }

}
}

when you edit system .lua files, all you really have to do is restart the matron process.

---

there’s one point of confusion. our system supercollier classes are named Crone... because in v1 of norns, the crone process did not exist, and all audio tasks were performed in supercollider. we tore all that out but didn’t update the class names. (i wish they were named Norns, NornsEngine etc., but now it almost feels too late…)

[tekknovator]

@zebra

(of course make sure you stop the norns stack running first! also ./waf -j 3 is useful.)

I did not stop the stack
Recompiled because I got carried away exploring the project. Also, I managed to brick the device (from ssh/maiden access at least) so I had to re flash my card Apparently I was too eager to break something.
Sounds like valuable tips. Gonna try that next time.

Also thanks for the drawing. Makes it a little clearer.

@zebra

when you edit system .lua files, all you really have to do is restart the matron process.

I settled for „editing via ssh, ;restart and output reading in maiden“ workflow.

I was wondering where all the logs go? Is there a file I could tail or terminal to connect to? Would save me a window switch

[ngwese]

@tekknovator

I was wondering where all the logs go? Is there a file I could tail or terminal to connect to? Would save me a window switch

If you are iterating on changes to the lua core code (and comfortable doing things in one or more ssh shells) the most direct way would be to:

• stop matron by running: systemctl stop norns-matron
• (re)run matron directly via: ~/norns/build/matron/matron

Doing the above will bypass the web socket based plumbing which exposes the matron REPL such that it can be used by maiden. When running the matron binary directly it is worth noting that there is no prompt to indicate that you can type in lua code directly. Once you want to put things back to normal either reboot or run systemctl start norns-matron

If you want to do all of this in a single window I’d recommend using tmux or screen to multiplex a single terminal window. I personally just use to multiple windows + ssh.

If you want a slightly more feature-full REPL experience in a terminal you can also try the maiden repl command. It has some rough edges (it pre-dates the web based maiden) but it does offer line editing and switching between the matron and supercollider REPLs - something which running matron directly doesn’t offer.

UPDATE: I’ll also add that a full rebuild with ./waf -j 3 will more often than not crash the system unless one stops all of the norns processes because the system will run out of memory (rpi3 / cm3 only has 1GB of RAM). I myself leave the norns stack running and stick to ./waf -j 2 or ./waf -j 1 if I’m doing a full rebuild. …after that I let waf use all cores (which IIRC is its default behavior). The heaviest component to compile is the C++ code which is providing Ableton Link support - once that is built incremental rebuilds are pretty quick. If the changes are just to the lua code then as other have said it is not necessary to rebuild matron.

[tekknovator]

So much valuable information! Thanks @ngwese

[tekknovator]

sent a pull request:
https://github.com/monome/norns/pull/1431

[tekknovator]

Testing midiport selection:

Testing Norns midi clock broadcast

Testing tempo change of two Follower devices in broadcast mode:

Midi clock broadcast tempo change test

[mattallison]

@tehn

It seems that there wasn’t a resolution on distributing clock to all attached midi devices.

There are a couple uses cases for example sending clock to devices that have no other midi i/o other than USB, for example an OP-1 and OP-Z, it would great to send clock to ALL devices from Norns if we can make this happen.

Regards

Matt

[tehn]

this is trivially easy to do within a single script (with a couple lines of code), but i want to confirm that what you’re requesting is a menu-driven solution?

[mattallison]

Yes, a menu driven system at a system level. Thinking under “clock” and midi clock with an option for “all devices”.