To my understanding, banana systems needs to be grounded together because, as said, the wire only carries the signal and no ground. This is due, I think, to the fact that the signals in the separate devices do not share a common reference/ground so they do not relate to each other properly. I’m sure someone could put that in more technically correct language!
The signal will certainly act weird without banana devices being grounded together but I’ve never heard of or experienced any damage happening to equipment as a result of this. The only time I think damage could occur is if the signal output by one device was outside the specification for another device. So, synth A outputs +/-10v CV signals and synth B accepts only +/-5v and does not have any protection on its inputs in the event of too large a CV coming into it. This isn’t exclusive to banana systems though and would be a problem with systems such as Eurorack which use cables carrying signal and ground.
There’s no need to ground separate Eurorack cases together or any other devices using signal/ground cabling in the same way as banana systems. The ground in the cables ensures there’s a common reference for the signals and all plays well 