Hmm thought norns was now using a 128 sample buffer, with 2 buffers on output side? In that case, synth-control latency due to audio blocks should be 4ms+/- 1.3ms (Max jitter being half a block, average latency being a block and a half)
Now, worrying about the best part of 2ms really would be dumb - except that latency usually stacks, and worst case jitter adds linearly. I would guess block size is probably not the latency jitter bottleneck of norns…