Scarcity and Precarity in Small Manufacturing

Not normal is now normalized: the last few years have been unpredictable at best. It feels time to share some experiences and observations of the state and future of small manufacturing.

Most people experience supply chain issues as simply not being able to purchase something, or only being able to do so at a greatly inflated markup. But another situation emerged that is even more frustrating: the potentially great difficulty of repairing existing things as a result of replacement parts either not being made or being unobtainable.

To lead with an actual case: we have not been able to get the STM32 microcontroller used in the Crow (and Grid) for well over two years now, and all estimates are “unknown” (the words from various sales reps I’ve spoken to, ie those at Mouser). Since we’ve sold out and also exhausted our backup-repair-stock, if someone contacts us with a hardware-related break-- for example, somehow the STM32 is fried-- there’s basically nothing we can do (that was previously standard procedure).

One resort is to salvage: when a PCB is sufficiently busted in more than one way, it enters the scrap yard (a particular cardboard box in our basement) where it can have further parts pulled from it. This was previously not economically worthwhile: the success rate of part swapping isn’t 100%. Frequently after substantial time has been spent lifting and installing, a “new” part would also be malfunctioning (likely when a PCB was subjected to “bad conditions”). So previously the parts would just go to e-waste since new components were widely available and irrationally cheap.

We are no longer in that era. So I’d like to instill this message, as strongly as possible: your machines are not easily repairable. They may be incredibly difficult (interpret: expensive) or impossible to repair. Please let this inform your treatment and expectations of the machines in your life.

It’s also potentially a good time to learn some repair skills. Soldering a DIY kit is a good start, but doing careful repair and salvaging is a fine craft. There’s a long tradition of musicians also being caretakers of their instruments (in school, as an oboe player, I was taught to make my own reeds, identify and fix leaks, tighten and lubricate various action… this is the same for basically most acoustic instruments)— consider this my call for a greater number of people to bring this attitude more widely to electronics.

Why not just put in more engineering time and use a different part?

This sometimes applies to a somewhat generic component (like a passive) but the industry now is mostly constructed using highly diverse, specialized, and non-pin-compatible semiconductors. (My frustration with this hyper-capitalist situation is the subject of another essay). For repairs, this is generally not an option.

For the production of an existing design it’s possible to rework the section of the specific part in shortage. This might mean changing a package size, moving signal connections around, adding/changing support circuitry, and/or changing firmware code. This invariably requires a full prototyping for confirmation which is expensive and long despite a large portion of the design staying the same.

And there’s no guarantee the new part won’t also go out of production, so the cycle begins again.

But of even more consequence: a single project with multiple designs adds a level of complexity which creates further problems for documentation, support, and maintenance. Recall this gem of a user experience:

We did this because the global supply of the pi 3 dried up (partially due to manufacturing crisis, partially due to basic planned obsolescence) and because the audio codec similarly was gone (due to a factory fire prior to the pandemic, which doubly made restock lag behind after the real crisis hit.

In addition to being confusing, this transition took a ton of engineering time— some of which was volunteer efforts by talented contributors who could’ve been working on more creatively impactful aspects. Support in service of the new confusion is likewise diverted from time supporting scripting education. This is the new cost of doing business.

Finally, for some parts (like the Crow’s microcontroller) there simply isn’t a well-matched similar chip, so a redesign isn’t really an effective option.

Won’t this all work itself out soon?

It’s hard to say, and most of what I’ve read and experienced suggests no, not really. Despite using predominantly domestic US manufacturing, we’re still hugely impacted by shutdowns in China due to the pandemic (which is not over, despite potential appearances). Fossil fuel prices are incredibly volatile (war/etc) and have a ripple effect out to every other commodity. Fires and floods and ice storms and hurricanes are all more frequent due to climate change, taking factories offline for potentially long stretches while aging infrastructure needs to be repaired (as happened recently in Texas). Political battles waged with embargo and import taxes (variations for each administration).

Skilled labor shortages also impact our small operation. We use a local circuit assembler. They’ve been experiencing a substantial amount of turnover during the pandemic and are frequently short-staffed. Despite their best efforts and the comparatively high costs we pay (versus an overseas assembler), quality has suffered. Which has created a load of in-house debugging and repair and costly send-back support.

As supply issues deepen it’s only more likely that small manufacturing becomes marginalized. Large corporations don’t buy capacitors from Digikey or Mouser: they go direct to the OEM. In many cases, they just buy the OEM, and have exclusive access.

Companies making expensive machinery (ie, heavy large tractors) have a pretty strong financial incentive to engage in edge procurement techniques. Such as, purchasing large quantities of new stock consumer goods (think, appliances) which contain a specific needed semiconductor, extracting that part for use, and discarding the rest. Acquiring a single part puts the remaining unused components into higher demand: a sort of avalanche effect (not to mention the huge amount of waste, since generalized recycling of e-waste still doesn’t “make sense” short of harvesting a small range of known high-value parts.)

It’s hard to see a return. More likely I’m expecting a sort of paradigm shift.

I don’t mean for this to be a drag. I think I simply need to share some of the invisible weirdness that deeply impacts this small universe. Written somewhat quickly without time to do annotations with references. I’ll attempt to edit this for clarity as certainly there are some issues/errors as written. Any suggestions or experiences or links to reading would be greatly appreciated.

217 Likes

I’ve heard a lot of both from optimistic and pessimistic perspectives on the shortage and tend to agree that this problem isn’t going to be resolved soon. As the resources needed to make these chips become more contested/sparse and the collection of manufacturing power becomes more centralized and politicized, I feel like this will likely get worse.

One solution may be to develop new devices and objects from recycled e-waste, there have been a lot of people getting into revitalizing old/obsolete architectures. Chip manufacturing is such a complex, expensive, resource intensive process that it’s doubtful we’ll have small-scale manufacturing any time soon.

How can we optimize for scarcity? I know the norns project was trying for this with the raspberry-pi and still ran into these problems… one thing we can count on, everything is transitory.

11 Likes

Reading this im really (i mean REALLY) happy i bought a crow prior to this.
Shits fucked up big time.

3 Likes

Not much to add, just to say - nice writeup! that’s very enlightening to the situation and I appreciate y’all’s transparency so much :blush:

I’m sure it’s not as long as most people hoped (and in the grand scheme/ideal world it’s awful), but in the world we live in - 4 years on the same SBC is pretty damn good, especially with a drop-in(ish) replacement now available(ish)

Love

-z

5 Likes

Most definitely!

I’m dreaming of a world where computing could be as stable as sand, wood, water… So much of computing has become a mess of unoptimized (but powerful) application layers which require more and more of the underlying architecture. Primarily to make things very general-purpose… Optimized single purpose computing with the lowest necessary hardware requirements is attractive to me.

A little off-topic but I love the idea of @neauoire’s Uxn project as an example of human scale, small computing.

18 Likes

I wasn’t aware of this, and it’s an absolutely crazy thing.

8 Likes

Thats where all those ps5’s ended up. In john deere tractors /irony off

3 Likes

despite the dire situation i always appreciate detailed updates like this and especially…

I was not careless before but extremely confident in your repair policy…i’ll do my best to avoid damaging any of it from now on

19 Likes

Witnessing this first hand at my work, where scarcity of various electronic components make everything super tedious and force redesigns and production delays for the last 1,5 years – i can totally relate.

thanks for your open words. Much appreciated

3 Likes

I think this is where the companies that open source their hardware and firmware can win people over - eg Mutable Instruments and TiNRS - if components are unobtainable the community can redesign relatively quickly. Also, with fully available designs it’s very easy to repair their modules yourself. It’s definitely part of my purchasing decisions now…

9 Likes

I’d like to share a little personal anecdote, the regulator connected to our alternator recently burnt up. It’s all resin encased and thus, electronic trash, unrepairable and unrecyclable.

max_charge

The companies making compatible regulator are now charging 600$ plus, with about 5-6 months of delay. We couldn’t wait that long, pay that much, only so it would die on us again in 5 years. The new models are equally unrepairable. Their diagnostics go only as far as telling you that it needs to be replaced or not.

We looked for a used one but in the end, we ended up replacing the whole thing with a 10$ ceramic reostat.

in hand

I’m not sure where this lands in the repair-reuse scale, but we figured that maybe learning what this thing did, and learn to do away with its automation, might be a net gain after all. The best amount of technology, is often the least amount of technology.

This obviously, doesn’t solve the raspberry pi supply chain issue, but maybe this gives option to someone trying to solve issues by throwing computing at the problem.

installed

Also, like, knobs, I mean… KNOBS. You get it.

49 Likes

“One Weird Trick Regulator Manufacturers Don’t Want You To Know…”

26 Likes

Trying to remember where I saw a recent toothbrush teardown that revealed a Cortex M3 STM32 in it or something bonkers like that…

4 Likes

Reminds me of this single-use digital covid test: Electronic Covid Test Tear Down Shows Frustrating Example Of 1-Time-Use Waste | Hackaday

2 Likes

As a builder of acoustic instruments who also offers very popular custom preamps for said instruments, I’ve been hit by this as well. I can’t imagine how precarious it must feel for a small manufacturer of primarily electronic instruments.

If anything, it’s increased my hoarding tendencies — if I had only bought 100 of the op amps I use a couple of years ago (something I could easily have afforded and justified, and have done with various other components and parts) then I’d have no problems for the foreseeable future. As it is, I’m stuck with enough for three, maybe four more units, and then I either have to hope that I can either make BJT op amps work in place of the JFETs, redesign the board, or try my luck with some of the shadier suppliers who claim to have a few thousand of them lying around.

4 Likes

20 characters of yikes.

Thanks for making this post, it’s been really useful for me to consider the impacts this has and will have.

While we work towards a beautiful solar punk future of increasing biodiversity and resilient, caring human systems, it’s important to note the non-linear nature of things: there is huge variance and things will get worse for a time before (hopefully) we can make them better.

In a way we have been living in a world of technological hyper-abundance, fuelled by the just in time delivery of components with historically staggering power.

In terms of physical effort a litre of petrol contains the same energy as someone working hard at manual labour for ~129 hours. (If you paid your petrol minimum wage how much would it cost to drive 100km in your car? L/100km x 129 x min wage. It’s $14,400 for me)

If we think of mental effort these chips and the systems they enable perform tasks that would take inhumanely long time scales for an individual to complete.

Now these willing servants become more scarce we experience the loss of their support and are exposed to the fragility of the system they enabled.

This precarious little ecosystem, which has become so precious to me in the year since I discovered it, sits at the fringes of industrial organisation and as such feels the effects of wider forces first and with unmitigated impact. The canary in the coal mine warning us of a threat before we feel it ourselves.

Tomorrow I will be fixing a 25 year old CRT monitor, built mostly with through hole parts, thinking how this hobby project embodies the smallest personal germination of practices which might become increasingly important.

I am committed to taking better care of my gear and the people involved in making it.

31 Likes

Thanks for those observations.

And thanks Tehn for the transparency in sharing the reality of the current situation.

If I had more time I would have liked to delve into how this is an indicator of the inevitable imposition of scarcity on those of us who have benefited for generations from global systems that have imposed scarcity and much worse on fellow humans who live “over there.”

Those chickens are coming home to roost, or whatever the metaphor is…. We’re all living in something like Jon Hassell’s Fourth World…

Thanks to a post on lines, I recently heard about solar punk…. It’s a vibe I’ve been feeling for 40 years but didn’t know that it had a name. I always thought of it as Love Punk…

Anyway, so much appreciation for this place and these folks!

11 Likes

Cory Doctorow published a semi-related article recently, which I really enjoyed. Well… maybe “enjoyed” isn’t quite right, but still…
https://pluralistic.net/2022/06/01/factories-to-condos-pipeline/#stuff-not-money

16 Likes

Sounds like I should consider a second-hand Teletype if I’m looking to buy one in the next few months.