passive will typically introduce impedance which may result in gain loss or unpredicatability depending on where the signal goes afterwards-- just a reminder.
the easiest thing you could do, for a an approximation of panning (this scheme would introduce substantial attenuation) is use a dual-gang potentiometer. it’s basically two pots hooked up to one knob. if you use a linear potentiometer (not an audio/exponential) you can feed the input signal into opposite sides of the pot, GND into the inverse, and use the center pins as L/R.
i must say, though. active circuits are awesome, and not much more complicated. an 8-pin dual op amp and a 9v battery is the next level.