Arduino grid study question

I’m using the arduino grid studies info and library with an arduino due and recent 128 grid (kit with ext5 power) and its all working nicely but I really want to be able to hot swap my grid between the arduino and my WW

When I unplug the USB the arduino keeps working indefinitely, brilliant, but when I plug it back in sometimes it works but often it crashes the arduino (unplugging and plugging into the WW works every time)

I’ve tried a few different arduino sketches, same thing happens, seems if I plug the USB back in within a few seconds its more successful than if I leave it unplugged for more than a few seconds

Has anyone else noticed this? Wondering if it might be a problem with my board or perhaps just not possible?