The core function of the system I use has always been the granular manipulation of samples. That’s the kind of sound I really like to hear and make - all those cool, complex textures. I didn’t know how I was going to do it, just that it’s definitely what I wanted to do. Exploring the how is the modular adventure/rabbit hole.
It started out as a Clouds and an LFO. Additions or changes have nearly always taken the form of considering what they would contribute to my granular goal. Since this is mostly all about external audio and samples, I have practically no interest in VCOs, programming or sequencing or such stuff. My interests are in continuous, gradual textural changes (Reflex LiveLoop, g0, Pitchshaper, Sloths), rhythms (Rampage, Two Princess) and FX (329 Phase/Flange, 2hp Verb). So after several years, the system still has the same overall orientation (even though Clouds is long gone). It’s a sample player and rhythmic texture synthesiser! It can output several independent voices at once, but I still rely on a DAW to round off the sound and mix with other stuff.
Although it’s sometimes been difficult, my problem mainly hasn’t been deciding what functionality and interactions to set up. It’s been moderating impulse purchases and budgets. 
Just noticed I’m replying to a post 22 days old. My bad. I just felt the need to relax and post something somewhere.