I read this a bit late, since you already returned the unit, but yeah, I also had high-pitched noise when I was monitoring/sampling. Just like you, it was occurring as long as “Line In” was enabled, even when no cable was connected. And even when “Gain” was at the lowest setting.
Counter-intuitively, I had to turn the gain up in order to fix my issues. I think the on-board limiter or something similar within Tracker was causing the problem. As long as there was enough normal / “not bad” noise present with the gain up, the “bad” high-pitched noise from the limiter would never kick in.
Also, I had been using a mono 3.5mm cable, I figured it wouldn’t make a difference since stereo would just sum to mono anyway. Well, I switched to a 3.5mm male-to-male stereo cable (this one, though any basic aux cable would be the same) and it also seemed to help significantly.
So between those two solutions I was able to record in with no high-pitched noise or weird limiter noise kicking in. Just a normal, acceptable (to me) amount of line in noise.
If you decide to try out a Tracker again, I hope one or both of those tips can resolve your problems.