I did some further debugging tonight. Unfortunately it raised more questions than it answered! I seem to be finding analogue electronics rather tricky
I’ll document my investigations here incase it’s of interest. I plan to go back to my breadboard tomorrow to try and reproduce the problem there. Here are some fun images from my Fisher Price oscilloscope 
Here’s a 440Hz square wave going into the input jack
and here it is after the first capacitor C1
I was not expecting it to distort like that? I would have thought a 0.1uF capacitor would discharge much faster than that, to only filter out very high frequencies.
Here’s the 5V line with the sensitivity turned up to 10mV. You can see a lot of noise, but the amplitude is reasonably small (less than 10mV), although it will get amplified by the output stage. I may try experimenting with a voltage reference like this http://uk.farnell.com/texas-instruments/lm4040ciz-5-0-nopb/voltage-ref-shunt-5v-to-226aa/dp/1564659