It’s about speed. If they’re all matched, you get theoretically 4x the bandwidth across the entire memory. If they aren’t matched, you only get the speed boost for the portion that overlap, and in many cases the system might just default to 2x across the whole thing - I’m not sure what the iMacs do in hardware, but you definitely won’t get the max performance from the memory.
32GB is plenty for me to be editing 4K video, I don’t anticipate for audio you’ll ever need more than that and that’s 4x8GB which is usually pretty affordable relatively speaking. I have a 2014 retina iMac (max specs) and it’s still fast fast fast.
That said, my 2014 13" (also loaded - 16GB RAM) is absolutely no slouch and I suspect the “display drivers” are not what’s causing any sort of audio related problems whatsoever unless your plugins are dodgy. I run Ableton 10 on it all the time and it’s rock solid and performs very well. I have zero plans to upgrade it for quite a while.
You could try a brand new, from scratch OS reload on your existing hardware (deauthorize all your plugins/apps first, and take a FULL system backup, then reinstall things one at a time as if it were a new computer, don’t restore from your backup other than to directly copy your data itself - not programs or configuration - over). I do that every few years – it’s not necessary, but I do seem to have fewer technical problems than folks I know who don’t do that, knock on wood!