It’s funny, the two Max for Live devices I’m currently developing and which will form the core of my performance tools are a step sequencer (tone generation) and a looper/sampler/shifter (for looping and mangling live input such as voice, guitar, and other sounds).
I really like your rationalization of the playing/listening dichotomy (or lack thereof), and it really resonates with other similar approaches I found emerging also when playing, for instance, a guitar through effect pedals. As you point out, one can play the effects as well as the instrument: sometimes the effects become instruments on their own accord.
Also about the complexity of sets, I completely agree: one should build muscle memory to enjoy performing with these tools. It’s way more fun and liberating than setting up parameters on a DAW, and it really feels like playing. I’ve been on stages a few years back, trying to remember parameter settings for multiple drum machine and synth channels changing for every song and I was way too stressed out to enjoy the gig. You shouldn’t have to think (too much) when playing, you know?