While that paper gets into the technical issues, I’d be interested in a more musical discussion about ways of collaborating musically in real-time over the internet that might be more tolerant of latency and jitter. If you can’t eliminate latency altogether, how might we work around it?
Of course, it’s always more fun to jam in person, but then again, we’re all already sitting together here in this big room called Earth…
ive always been interested in firing up an certificate based openvpn server for “players/participants” to input and receive OSC messages that are routed to another master machine that controls musical elements, video, or lighting elements much like @Rodrigo experiments. Or to play a show where the first 10 attendees receive a android device connected via wifi actually get to participate in the show by interfacing with simple touch osc layouts (got the idea from the beastie boys performance where they handed out video camera to the first fans to arrive at a show).
The latency could be “quantized” by rounding up/down in time so the low level parameters would still “feel” the groove. OR DONT! see what happens!
edit:
This is a fucking brilliant paper. RTP is super fast, ive had experience with nomachine NX. More than remote desktop… we’re talking remote usb devices showing up in our dmesg or lsusb as a REAL device IDs.
edit edit:
this is also fantastic. provided everyone on the same routing subnet, we could have one DAW and the entire community’s grids talking to its serialoscd server!!! maybe…
root@inmymind:/opt# apt-cache show usbip
Package: usbip
Source: linux-tools (4.3.1-1)
Version: 2.0+4.3.1-1
Installed-Size: 129
Maintainer: Debian Kernel Team debian-kernel@lists.debian.org
Architecture: amd64
Depends: usbutils, libc6 (>= 2.7), libudev1 (>= 183), libwrap0 (>= 7.6-4~)
Description-en: USB device sharing system over IP network
USB/IP is a system for sharing USB devices over the network.
.
To share USB devices between computers with their full functionality,
USB/IP encapsulates “USB requests” into IP packets and transmits them
between computers.
.
Original USB device drivers and applications can be used for remote USB
devices without any modification of them. A computer can use remote USB
devices as if they were directly attached.
.
Currently USB/IP provides no access control or encryption. It should only
be used in trusted environments.
.
This package provides the server component ‘usbipd’ and the client tool
‘usbip’.
Description-md5: 84110e083527a27b37417dcab69e5ab7
Tag: implemented-in::c, role::program
Section: admin
Priority: optional
Filename: pool/main/l/linux-tools/usbip_2.0+4.3.1-1_amd64.deb
Size: 41346
MD5sum: 16adcb3214cde01475ab9344968d77a7
SHA1: 206b87a25b61a0d2c6ab876a68022935e6573917
SHA256: bf6c4a657962408d890b73d7a13904351d12891a9f7a1d3ebb675109355b240f
@raws you’re way ahead of me but I’m happy to see this idea might have some legs. I’ll need to noodle on this some more to really grasp its potential but in the meantime I’m up for testing any scenarios folks might come up with.
Paper looks interesting. The tech side of the implementation (in the newer version of dfscore anyways) was looked after by someone else, so I don’t have technical insights there.
As far as musical applications, in most of the contexts I’ve used this kind of approach latency has been less of an issue since it’s been largely unidirectional communication (e.g. computer to performers), so latency wasn’t an issue. Even if it was 500ms latency, everyone had the same latency, so in real-time performance, nobody had latency.
For pieces where feedback was involved, it’s more complicated. Thankfully I’m working in more open musical forms, so the perceptual boundary for togetherness isn’t the same as in a tight/synchronized electronic music. There, 20ms give or take is on the edge of being noticeable. With acoustic instruments performing together the “tolerance” is higher than that.
On a more conceptual/aesthetic note, I’ve been getting into the idea of “agency” as a manipulatable parameter, creatively. For example, I’ve been working on this improv analysis stuff for a while now and part of the future development of that idea will be to use it to generate dfscores which are related to your improvisational tendencies in either a complementary or uncomplementary way. Or anywhere in between. So using analysis data from myself (or others), I’d be able to play with “agency” itself with regards to how much me vs my tendencies vs the opposite of my tendencies vs other’s tendencies etc… is present in a given improvisation, or improvisational prompt.
So them’s some of my thoughts on the matter, hehe.
Love it! Part of what makes latency “matter” more in this scenario is the fact that I’m imagining us jamming over the internet, rather than using networking in a shared physical space.
I think it basically means using musical units (“clips”?) no smaller than about 50ms, to allow for the quantizing that @raws refers to. But that’s just accounting for network and device latency. There’s also the harder to measure cognitive latency that comes from not sharing the same physical space. This could be mitigated somewhat by video conferencing, but that puts more strain on the network in a situation where network strain is highly sensitive.
So there may be good reason for requiring much longer duration musical units, not just to allow for physical latency, but also to allow time for collaborators to grok and react. @Rodrigo I gather your post is really relevant to these issues, so I’m looking forward to chewing on it, thanks!
And it looks like my intuition might have been near the mark:
Since the inherent latency of the Internet prevents true realtime synchronization
of the jam, and playing with latency is weird (and often uncomfortable),
NINJAM provides a solution by making latency (and the weirdness) much longer.
Latency in NINJAM is measured in measures, and that’s
what makes it interesting.
So, what NINJAM does is compresses audio with Ogg Vorbis and sends that around. That’s really interesting and it leaves the situation wide open to the use of any kind of electronic or mic’d acoustic instrument. That’s appealing!
That being said, I grokked enough of what @raws was talking about to sense the potential for other directions this could be taken as well.
Anyway, I think I’m going to try setting up a NINJAM server on Digital Ocean, and see what I can see.
Edit: I’m wondering if dfscore and NINJAM might be good buddies for this project…
Cool to hear ninjam is still going. I got away from the monome for a long time and hadn’t thought about it since picking it back up. Maybe I’ll get on there again and see how it feels.
A system with hardly any latency would be sweet as well.
The technical aspects of it all are far beyond me unfortunately.
Yeah pushing audio across a network really complicates things, especially since it’s bidirectional (so all latency is multiplied by two, for a roundtrip).
So solutions like having it be a musical relatable amount out of time makes sense (in that kind of music).
If clips are known ahead of time, though, you can just have all assets on each device/node and only push timing information around, which has a significantly smaller network footprint.
With NINJAM, you get audio, and that’s pretty much it.
I can imagine far lower latency (but still no less than about 50ms at best, 120-130ms is possibly more realistic, especially when you consider the global nature of this community) with OSC or MIDI messages. It adds a bit of complexity because your sound generation now needs to be local. This could be made less cumbersome if players chose to constrain themselves to free sound generators (we all love Max around here, right?) And yes, pre-defined clips dramatically simplifies this whole idea. (But maybe removes some creative potential?)
But I’m going to play with NINJAM first, simply because it’s there and why not. Then I’ll start digging around for latency-compensating MIDI or OSC musical jamming stuff (to see if such a thing already exists). And read @Rodrigo’s piece and finally get around to getting my hands dirty with dfscore, and listen to the rest of this FSOL recording… an enjoyable evening!
Haha, well, helps if you slow down and bother to read the dates on things. the OS X client is circa 2005 and only supports PowerPC architecture. So I suppose I was rather premature with “NINJAM is still around”. Back to the drawing board!
Edit: hmm:
Edit 2: Well shoot. “Transmit” is disabled in that Chrome extension. I’m gathering there are more up-to-date clients floating about, but it’s not clear if any of them are canonical. Things seem to be in a somewhat fuzzy state at the moment.
Quintet.net is an interactive networked multimedia performance environment invented and developed by composer and computer musician Georg Hajdu. It enables up to five performers to play music over the Internet under the control of a “conductor.” The environment, which was programmed with the graphical programming language Max/MSP consists of four components: the Server, the Client, the Conductor as well as the Viewer add-on.
The players interact over the Internet or local networks by exchanging musical streams (control messages) via the Quintet.net server. For this, various inputs ranging from the computer keyboard, MIDI controllers, sensor input and/or the built-in pitch tracker can be used. On the server, the streams get multiplied, processed by algorithms, and sent back to the players as well as to the listeners. In addition, a sixth participant, the conductor, can control the musical outcome by changing settings remotely and sending streams of parameter values either manually or by utilizing a timeline.
The environment uses two network protocols for exchanging data: OpenSoundControl/UDP for time-dependent events, as well as TCP for safe data transmission. It also uses a mechanism to compensate for network jitter.
Quintet.net’s open architecture accommodates various outputs such as the built-in sampler, MIDI as well as VSTi instruments and custom designed software patches for instrumental playback. It also features granular synthesis controllable by the players and the conductor.
Quintet.net’s notation layer allows for better interaction and control on a symbolical level: The performers see the music that the participants produce on screen in “space” notation on five grand staves. The conductor can also send musical parts, which are displayed on screen and played back by the performers–all in real time.
The Viewer add-on allows the realization of full-fledged multimedia pieces by taking advantage in a modular environment of the Jitter video processing capabilities. Thus, users can easily add new processes.
The server also implements a sophisticated mapper/sequencer which allows to map any input to any output, be it musical or visual.
The music performed with Quintet.net is a combination of composed and improvised elements. The lack of real synchronicity due to the usual delays on the Internet, necessitates the adaptation of a genuine “Internet” performance style for which John Cage’s number piece could be considered a model: Cage requires certain notes or phrases to be played within “time brackets.” It comes as no surprise that one of the first pieces realized with Quintet.net is an Internet version of the 1988 composition “Five” for 5 performers.
Local network performance, of course, can be played with unlimited precision.
Ah, rud ran that one - I participated in quite a few jam sessions, it was fun.
The original standalone NINJAM client is quite old and unmaintained. REAPER still ships with a NINJAM plugin that’s a bit better - and there’s a bunch of public servers up and running for it.
Jammr is probably the one that’s the most maintained and usable at this point, they’re running servers as a freemium sort of service (pay money if you want private sessions and access to multitrack recordings of sessions, otherwise free). The software behind Jammr is Wahjam, which is open source and a fork of NINJAM.
ha, i remember him doing this on the ‘fantasma’ tour in '97! except it was an sp202 or something…
as for network latency, i think we just need to work on telepathic improvisation.
failling that, long latency is very interesting in itself, it can really encourage a different pace of listening / responding, vs the sort of instant “reaction/imitation” process that seems to come up a lot in free improv, and usually takes discipline to resist.
this was the state of the art in 2010…no idea what’s evolved since then but I think Mark Cerqueira and Jascha Narveson would know…I’ll try to point them to this thread.
Haven’t done much further after 2010 and my work was only on local networks where latency is less of an issue compared to over-the-Internet (Internet with a capital i) latency.
The problems PLOrk (Princeton Laptop Orchestra) were having with local network latency at that time was (funnily enough) solved by just switching away from the Apple Airport Extreme. The PDF linked in ingMob’s post shows how a few different routers stacked up.
The background section of my thesis goes into some of the literature (up to 2010) of how people work with and around latency. The Stanford-Beijing joint performance is one of the more interesting ones where things were setup so the tempo matched the RTT between the two performance locations.
At the first SLEO conference a few years ago at LSU in Baton Rogue there was a performance from people over 7 (I think) places around the world. I forget the details, but whoever put that together would likely also be a good person to talk to if you are interested in over-the-Internet performances.
I can imagine there has been more work done in the past 5 years in the field. Hopefully someone who is still deep in the game can hop on and comment.
