I’m not sure I’d put learning programming in the past tense for me yet, but I’ve been hacking away at it (pun 100% intended) off and on for the past couple of years, and can definitely relate to OP’s relationship to it. No formal education, just an itch I always wanted to scratch and could never quite scratch enough it seems…
I’m a formally trained musician through and through, and like OP would hate to take on more debt to study formally (especially after the cost of music college). Also, it’s not so much that I’m thinking about a career change – it’s more that I’m looking at programming as my next step as an artist, just following my nose, so to speak, and whether or not it feeds my income in any way isn’t a huge consideration. Though, I admit that knowing that I now have a skill that’s actually marketable is pretty cool! 
It started by developing an interest in Linux for some reason. I’m not sure how or why. But I just thought the whole idea of Linux was unbelievably cool, and despite its massive adoption in so many areas over the past couple decades, I immediately realized that its potential had only begun to be tapped. Naturally, liking the idea of putting together my own OS led to wondering how an OS is put together in the first place…i.e. code.
At the same time, I suppose I naturally fall into the roll of a programmer since I’m lazy – by which I mean that if it can be automated, it’s a crime to not automate it. I’m a game/film composer primarily, and the amount of things in the pipeline that can screw up and waste unnecessary time is ridiculous, so I’ve gotten in the habit of automating a lot of the more dry/boring parts of my workflow, first with training wheels like OS X’s Automator, eventually leading to scripting languages such as Python and Javascript. It was messy and piecemeal for like a year, and honestly it still sort of is, but I actually like learning in such an organic way, driven mostly by need. It’s fun, and creative in its own right. You pick up a lot of interesting things when you allow yourself to tumble down a rabbit hole.
Now I’m living pretty deep in Max world, and as soon as some projects are finished up, I’m going to tackle C++ in a serious way. Screw building your standard EQ that tries to sound like a piece of hardware that already exists – I’m way more interesting in processing audio in ways no one’s ever thought of!
Maybe the best thing I’ve learned about programming is that the language really, really doesn’t matter, and the more languages you know, the easier it becomes to pick up another. It’s like learning scales or modes or something – it’s just a way of organizing concepts and achieving certain results. There’s more than one path to the top of the mountain, and as such there’s more than one programming language, but also most paths (and also languages) have a whole bunch in common.
For me it all just comes down to the fun of trying to solve a problem. It’s much closer to creating “art” (whatever that word means) than people generally give it credit for. In fact, I’m not convinced they’re at all separate…maybe programmers and the like are just artists that somehow convinced the world they needed to be paid!