To best honest I would not count on it. From my observations people treat concerts as social situations so the more the people the more the chance that there will be talking 
If everything will be already connected etc. then I don’t think that 10 minutes will be too short especially that there will be a lot of others duos playing etc. I think that between 10-20 minutes is good. We have an event here where people are grouped into trios etc on the spot by lottery as in your idea and we have 20 minutes limit on performances.
And I really like @ether advice to "encourage people to collaborate by listening to each other and responding. a lot of modular cats have a tendency to fire up giant walls of sound and try and dominate (which is fun in its own way). ". That is one of the reasons why I stopped going to more traditional jams because almost every time it ended in bunch of local guitarists playing never ending solo. My ideal jam would be where people create songs on the spot and I think that encouraging people to collaborate is a good way to achieve that.
Also some extra: if possible try to invite some VJs to the improv or take into account lightning in the space that you will be playing to create some ambience. I think that I can really change the perception of music especially if it is more abstract as modular music tends to be.
Edit: and about “Prepare for a mass modular improv orchestra”: from my experience it is good to have someone that will act as a conductor in such orchestra. Musicians that I met have tendency to play all the time and it can get really out of hand if more than 4-5 people are playing.