I think you’re already working around the issue by the sounds of it. In this case, the Mangrove is responsible for the majority of the pitch shift, not the teletype (though there are tiny offsets on the TT outs). This is a due to a difference in internal ground potentials that I didn’t understand 3 years ago when I designed the module.
In general it’s not a problem, but you need to do the calibration with the cable attached (0v is CV 0 V 0 not cable-unplugged). The only thing that matters for calibrating the module is the difference between the voltages. V1 to V3 would work too, but it will make no difference. Which TT output doesn’t matter, any slight difference of offset will be represented at the higher voltage output as well.
In practice, if you’re trying to do tightly tuned sequences, i recommend you tune & prepare your patch with the volt/octave source pre-patched.