I think the best advice to keep music making cheap is to buy less, and overall, to do with less. Depending on your budget and goals this might be done very differently. For some, it’s buying 1 piece of kit of high quality, and pretty much nothing else. For others, it will be to buy cheaper alternatives.
Second advice would be, if you can make your music with a computer, that will be cheaper. And I would +1 @lloydcole advice of iOS in the same philosophy 
But if you want to buy cheap gear:
- Look for previous versions on the used market(ie mk1 instead of mk2)
- Look for less featured versions of gear (model:samples instead of digitakt, el capistan instead of timeline, etc)
- digital reissues of old classics