The SP404 could work for you (definitely want the SX or the A for this purpose), but it is limited in some key ways.
It can play back a number of samples simultaneously, I think it’s 12 voice polyphony, but I also think stereo samples might count as two voices, not 100% on that. Lots of fx and many for them are good, but you can only use one FX algorithm at a time, though you can choose which samples are routing through the fx with the “remain” button. Also worth noting that you cannot repitch + change sample speed simultaneously on the SP, just pitch shifting effects. & the time stretching is pretty bumpy. These are interesting sounds but not necessarily what you’d expect from a typical sampler.
You can definitely play the stems back live as long as you’re comfortable with your timing, and as long as you don’t hit the polyphony limit. You can also sequence the stems but I’d absolutely look into the patterns sequencer workflow if that’s your preference, some people really dislike how it works, but it does offer long pattern lengths.
EDIT: realize I was assuming a lot about how groove based your music is, I will say having the long loops to free-form swell from one sound to the next is a wonderful sp experience, too.