this comment from @mzero opens an important topic.
i don’t know if we have a language for composition of control in this sense. mark, i think you are in a good position to help guide the development of one, which will be a shared effort - we are, in fact, all mere mortals and the first round of instruments for norns are likely to be somewhat ad-hoc, fitting limited roles.
there are so many ways to go with the development of a deeper language/API. i agree that lua isn’t the most helpful language for building architectures. but it is flexible enough to be usable in many different paradigms, while also presenting a simple procedural face to the end user. (it’s also fairly efficient at runtime.) functional composition and currying is possible. it does lack a sophisticated type or class system, but there are options built on top of it.
(this isn’t meant to sound defensive - just have been thinking about it for a while.)
it would be interesting for me to see some concrete ideas of what an API for control composition would look like. the beginning is a good time to think about it, as we begin taking our POCs and tests and considering how best to structure them into more complex and useful projects.
but also that nothing specifically supports it (yet) and that the task is left as an exercise to the community?
So, we could use some design proposals. I’m curious to get the warts out of the way first, as stumbling blocks can also be waysigns pointing to a better architecture. On that note, I’d like to hear more about
What is it about Lua that makes this less than helpful? What types of things are helpful?
I’m also curious to know if you have any specific examples of composition of control done really well.
The thing I’ve been struggling with in recent years (professionally) is that I’m swimming (drowning?) in a GUI control environment where everything feels ridiculously overengineered, laughably brittle, and extremely error prone. (React Native mobile dev, 'nuff said). So, the longer I do this professionally, the worse the examples I’m surrounded by make me feel.
So I share your concerns (deeply) but I’m not confident about good patterns for achieving your goals. Let’s talk about good examples you know about and how we might pursue a similar approach in context of norns?
i should also be a devils advocate and say that i also think “ad hoc” is important and i love to see a ball of spaghetti that goes from A to B and just does what it does. (patches will always do this.)
I too appreciate the immediacy of ad hoc design, but I’m also a fan of more composed architecture. I’m really interested in finding out if @mzero has a proposal here. It’s a wickedly difficult problem that has dominated much of my adult life, so I’m on the edge of my seat.
That being said, I don’t think it’s strictly “necessary”. Lua and Supercollider are mature languages with huge developer communities, so it’s not like we’re starting from scratch with language design.
@zebra - First, let me make clear that I respect the amazing engineering, and appreciate the “launch and iterate” approach. Your interest in the area, sparking this thread, and acknowledgment that it has been on your mind, makes me like norns even more. If you’re going to play a unique instrument, it helps to know where your luthier is going!
Given the engineering trade-offs involved, Lua does seem like an understandable choice. As the language has only minimal abstraction mechanisms, and an unfortunate propensity for globals, it is easy enough to build a plate of spaghetti. But, with careful thought, we should be able to assemble something more layered (lasagna!)
But enough of Lua - it is what it is…
Indeed, @jasonw22, composition of control has been a vexing problem both in music as well as in general programming.
In music systems I’ve used and written over the last 40 years - most have completely given up when it comes to control. Think of all the power of ChuckK… and it gives you three byte MIDI messages for control. Max & Pd are much the same. The only system that comes to mind that has done anything more ambitious is Kyma.
Outside of music, the most applicable work I’ve seen is from the area functional reactive programming, where event streams can be combined in useful ways you can reason about.
I don’t yet have specific ideas for norns yet. But, some things I think about:
My examples from my comment were chosen carefully to demonstrate several kinds of control composition: Union, Wrapping, Layering, Multiplexing, Hierarchy.
There was mention of a parameter system - how would parameters compose as controls connected to them compose?
How do engine parameters fit in?
The user will need to map at some “outer” layer: One user has a keyboard controller with faders and buttons to map… another has separate keyboard and knob boxes. How does this layer of mapping differ from other control compositions?
I’m a huge fan of intuitive mapping such as the manner in which Ableton live “learns” the mapping by clicking the parameter and then moving the desired control. This leaves a lot of room for further engagement though: range, scale, macros, and context all dynamically change the meaning of any given mapping. To me, the important aspect is that any control that does more than one thing has a clear representation. For this reason I feel that controls in the physical sense should generally map 1:1 to a conceptual object. If one wishes for this to have multiple effects or to have contextually different interpretations, this object could be a macro (grouping) or other (user defined behaviour) object, but at least at the outer layer this translation needs to be as clear and uncomplicated as possible.
Complementarily, this can combine with predefined macros and interface object which do common, clever, or tricky things that the user can immediately grab and play with. So this layer critically needs packagable abstractions that can be shared and edited. I see this as the logical patch bay, if you will. From here, the bridge between the physical and the digital controls can be routed, arbitrated, reacted to, etc. It’s a layer of flow on top of the layer of numbers and parameters.
Obviously more abstractions can be made. But this three-layer logical paradigm can handle a huge variety of complexity without becoming overwhelming, and can even enable exploration and creativity if the friction between moving and repatching components is minimized. Again, these details are the real work, but I can’t stress enough the need for his to be a physical, hands on, intuitive process as much as a descriptive, logical one.
controllers generate a stream of low level events (“button x,y pressed” / “encoder moved” etc). this event stream is then processed by the event processor which maintains its own state so it can detect higher level events, gestures. it then communicates them to the instrument engine which is not aware of controllers at all, it simply exposes methods to advance it to a different state (which could be as simple as “play a note” or “advance to the next step”). as the instrument engine transitions into the next state it can also generate events (“state changed” or more granular events) which the event processor uses to update its own state and provide visual feedback back to the controllers. (i’m not mentioning the sound engine here which should be also a separate layer).
that’s what i’ve been considering for a different project with a different main goal, but one of the goals is separating those components into their own layers, so you could easily replace one controller with another (for instance, instead of using a clock trigger input to advance sequencer replace it with a keyboard spacebar press).
this is mostly intended for static mapping (which would also take care of proper scaling etc) but in theory this could also be dynamic where the event controller changes the rules and remaps based on its state. this is one of the interesting aspects of tt/grid integration, as the UI itself is scripted, so in theory you could have the UI change based on the music itself. or have UI steer your playing technique a certain way as you play it. are there any instruments that do this? imagine piano where keys become “stickier” the slower you play.
This is something I’ve been thinking about (back burner) for a bit, as I (too) am a big believer that the mapping/control is the thing.
One mini solution that I’ve been toying around with, to facilitate similar ideas/problems in my coding is to pre-bake every module with several mapping schemes. Specifically 1, 2, and 3 parameter macro control. So if in a given context I want to control the whole module with a single parameter, I tap into that control stream, or if I want something more granular (but still ‘above’ the individual parameter level) I can tap into the 3 parameter version.
It’s a simple enough idea, which I think would be immediately useful, but at the same time it would be a shame to limit the scope of expressive output of a module by just guesstimating a mapping (even if scaled/tweaked one-to-many).
Towards that end, I’ve been casually looking at some machine learning stuff, with the idea that you could set it on a module and have it create a perceptual mapping space (based on the module-level exposed parameters) which can them have a network of connections made for 1, 2, and 3 (or more) macro controls.
My understanding of ML is the big limitation here, but this is the relevant stuff I’ve been looking at:
"Interactive computation of timbre spaces for sound synthesis control” from 2015 (http://stefanofasciani.com/?page_id=189).
And a bit of text from a friend (Owen Green, part of @tremblap’s team on FluCoMa) on the links above:
Note that he’s using synths here, which is a bit simpler: the ‘timbre space’ of a synth is fully captured by searching through the parameters, whereas the timbre space of a processor is a less predictable function of process and material. Nevertheless, what he does seems sensible, by using a lightweight supervised method alongside some descriptors to try and come up with perceptually smoothish mappings across parameters.
Please elaborate on this. What unique constructs for composition of control does Kyma have? I have zero experience with it.
To me, dataflow languages (Pd, Max) are based on reactive programming. Ironically, a lot of what is brought up as newthings in the hype around functional reactive programming (RxWhatever, Elm, all the hip javascript stuff) is stuff that multimedia artists have been working with since the 90’s. Pd and Max are of course not purely functional in the sense that Elm and Haskell are, but: everything is streams of events that can be composed in different ways: ie. filter a stream based on some predicate which in turn is based on another stream, or join streams according to some logic. There’s constructs in dataflow languages to work like this.
Aside: SuperCollider has a streams based (… impure-ish) Patterns framework based on declarative metadata that touches on many of the points you’re bringing up here, from a score perspective.
Wrt norns - I’m involved but i think this ship should leak from the top before release. All i can say is it’s architecture is not complex at all. Ideas similar to yours have been discussed and are possible to build as layers over the basic layers.
…
And… Lua and its
… oh yeah, agreed! Can be worked around, though, at least to a certain extent.
@Rodrigo makes a few good points here, as usual If the idea of mapping as metaphor, the many-to-many concepts, and mapping as expressive devices, interest you, I recommend browsing critically the nime.org paper archives. My favourite papers in there are in the reference part of my own paper - when we tried to tame content-based browsing of corpus (in CataRT) by recycling my ‘virtuosity’ on my good old mastered interface (aka a bass guitar).
That said, on the more general OP thread, any composed interface will be biased towards a musical outcome. This was McCarthy’s main reply to people saying that SuperCollider sounded better than Max. Since at the time they were both processing at similar rate and depth similar opcodes, his hypothesis was that language affordance was biasing the sound design in different directions. Since then, a lot of HCI research in ergonomy of software has confirmed this
I’ve been thinking about using the RGB pads on Linnstrument to indicate interval distances from the pressed note using color. Some early experimentation leads me to believe it can be a useful tool for learning harmony. That’s not really a very creative example, but it’s a very pragmatic one. This is a great train of thought!
We have been using some transparent ink over FSR led on a root / key locked system when designing Dek.Velocity sensitive (1ms when leds are de activated, 2ms with leds functionality) and poly aftertouch implemented
yeah, that’s a valid example. expanding it it could analyze what you played previously and use an algorithm to suggest what notes to play next by highlighting them, with different brightness indicating note “expectancy”.
I would assume a valid ‘algorithm’ in western music would be defined by notes belonging to same scale.
I am curious to hear what would you define as note “expectancy”?
Using dissonance and consonance is a common way to drive the melody toward the tonic center, however In traditional harmony the key does not define the tonal center.
My concern with note “expectancy” is that it is a sloppy territory.How would you define priority of expectancy in real time? The notes you have played before a certain moment in time are somehow irrelevant (i.e. Jazz Progressions)
It’s definitely not deterministic in terms of a single note or chord that you will definitely play next. It’s more of a menu of weighted options. Expressing that “weight” is where varibright would be helpful. I wanted to reserve color for note distance, but that’s for learning chord shapes on a grid controller, and that’s a more basic learning moment. Perhaps once you’ve learned the chord shapes, you move on to progressions at which point color is repurposed to indicate degrees of consonance/resolution for next note/chord.
Many specific details would need to be worked out in the implementation, and I’d expect several tradeoffs (or possibly options/toggles) would need to be addressed.
And once we did this for jazz, I’d also want to do it for classical, and also for non-western music (which would express a completely different set of expectations)
If we want to continue to discuss the “interface as theory teacher” idea, maybe it needs a new thread?
it’s just a vague notion - it would be up to the algorithm to determine which notes it considers to be more or less “expected”. the main point here is that it would introduce interaction between you and the instrument, where you decide how much you follow “suggested” notes. since the notes you play with also influence the algorithm it creates a relationship between you and the instrument, so you’re not just playing a sequence but interacting with the algorithm.
If the idea of mapping as metaphor, the many-to-many concepts, and mapping as expressive devices, interest you, I recommend browsing critically the 