I tried to do my PhD in this kind of direction. I got to the end without ever getting to the music part, but I got as far as applying game theory to turn taking (a kind of rhythm I guess). Game theory applies to groups of agents, so it’s probably best applied with groups of simulated agents that are interacting in some kind of improvisation.
My thinking lead in the direction of simulated robot drummers learning to produce music together. You have figure out how to get them to learn something musical. I invented a metric for turn taking, studied how agents learned to do turn taking and then described how to design agents’ reward functions such that they would learn to produce arbitrary, predetermined sequential joint actions. The next step would be to build that idea out to apply to music making robots.
I think the interacting agents paradigm is probably worth a digital eurorack module or two. In another chapter of my life, maybe.
I guess part of the problem is that music is generally a cooperative exercise with lots of coordination and a lot of the fun of game theory is for competitive games and games played without coordination. So maybe something like rap battles would be a good place to start?
I’m sure I could write a lot more here if I thought about it some, and dug through my list of paper references.
PhD thesis available at http://peter.raffensperger.org/Peter_Raffensperger__Measuring_and_Influencing_Sequential_Joint_Agent_Behaviours_(one_sided).pdf