I have this problem too. I feel like the encoder could still be used to select a favorite, but perhaps the “threshold” should be set a little bit higher?
My unintended favorites are often grouped together, giving me the feeling that this occurs simply due to my clumsiness.