I was also concerned.
Q: Does it constantly discharge even when nothing is plugged in or is the connector essentially the switch?
A: It continually discharges. It takes about a week to 10 days to fully discharge.
Q: What happens when the battery is no longer holding a charge?
A: Unfortunately the battery is not servicable. However, you will still be able to power it via the USB port and get the same quiet, hum-free operation.
It seems more like a studio item somehow… but you could potentially permanently install it into an instrument for recording/live performance. Maybe you could have it constantly receiving USB power when that instrument needs to be amplified.
I personally think it’s fine this way, with it discharging… because I plan my sampling sessions ahead of time, and I like to be mobile as possible to work quickly.