Are you talking about audio latency or MIDI->Audio latency? They’re somewhat different beasts, but in general plugin latency compensation only applies to playing prerecorded data from a track through a plugin (including the external audio plugin) for the purposes of that audio hitting the mix bus / output of the track at the correct time.
Most plugins accurately report their latency, so again, in most cases you’ll want to leave this at 0. For the External Audio plugin, the I/O of your audio interface is already taken into account, so this value should only be the latency of the external processor itself - for instance, if you’re piping a track through an external reverb with a measured latency of 2.6ms for a given algorithm, you’d want this value to be 2.6ms.
For external MIDI, my note above about sloppy MIDI timing (in general, not just Ableton, due to both USB MIDI implementation as well as the vagaries of MIDI itself) mean that you’ll get within a ms or two at best case. In this case, the latency figure should be the delay between the unit getting a MIDI message (e.g. Note On) and the first cycle of audio coming out as a result of that message. For instance, if your Digitone takes 1.7ms after receiving the MIDI note on to begin producing audio for that note, you’d set this to 1.7ms
In both cases, this will ONLY take effect during playback of previously recorded data in order to align it with the other tracks. It will not affect the recorded position of the audio or MIDI datastream. What it literally does is send the data for that track to it’s destination that much sooner, so that after it’s gone through the unit that introduces the latency, it comes out the other end in time-alignment with where it’s “supposed” to be with respect to the other tracks. Obviously when recording there is no point in adjusting the incoming data (except as we’ve discussed previously for the input latency of the I/O device, which is handled separately), and plugin latency, which only applies after the track audio is inscribed at a specific point in time, cannot move input audio or MIDI backwards in time to solve the latency issue, so it cannot be expected to make any difference whatsover when the track input monitoring is active or the track is in record mode.
In the MIDI case, the usual MIDI jitter of up to a few ms is quite normal and cannot be eliminated or compensated for, although it can somewhat be minimized (but that’s not a subject for this topic, since it’s not germane to Ableton Live specifically).
It’s not unusual for effects units to have different latency values depending on their settings, and even for some synthesizers to have slightly variable latency as well depending on a variety of factors, but usually this variance is quite low - on the order of 1-2ms total, if its there at all, and it should be fairly constant for a given patch or preset.
Does this help explain some of the issues you’re seeing and clarify some of your confusion around these various latency parameters and what you can and cannot expect from them?