64 Comments
User's avatar
TheElectricPilgrim's avatar

Well it’s not like software is critical for modern infrastructure and services…… hang on, wait I was channeling some politician/tech CEO inner monologue there. The world is nuts. Great article, thanks. I’ve seen AI work really well in some repetitive high-volume jobs like doc processing, but I’ve yet to be convinced it’s a good jack of all trades. It’s a complicated technology and too often the C-suite buys into the marketing hype while the engineers are left wondering why the tools they’ve been told to use are useless.

Denis Stetskov's avatar

The doc processing point is exactly right. AI is great at specific, bounded, repetitive tasks. The problem starts when the C-suite sees that demo and extrapolates to "so we don't need juniors anymore." Two very different conclusions from the same technology.

RNDM31's avatar

Waaaay back in the early days of Imperial Japan when they ported over much of their stuff from Tang Dynasty China, one of the imports was the quite sophisticated Chinese siege artillery and engineering know-how.

That was lost wholesale in one generation after an Emperor decided maintaining the infrastructure of specialist artisans and engineers was a superfluous expense and those folks duly found other livelihoods.

Fast-foward about a thousand years and you had the newly established Tokugawa Shogunate p much renting cannon and gunners from passing European traders to help reduce particularly stubborn rebel strongholds...

Denis Stetskov's avatar

Lost in one generation. That's the part people underestimate. Not decades. One generation of 'this is a superfluous expense, and the capability is gone.

Dan Elbert's avatar

Yeah, I suspect that software companies saving money by not hiring juniors is like farmers saving lunch money by eating next year's seed. Or US companies losing production capacity by outsourcing to China, etc, etc. But the system incentivates short term gains, so this is what we will get.

steak's avatar

What is hilarious here is that the knowledge and experience and skill and vision presented here is not even remotely valued in the West. The culture sees someone who works harder than they need to and who is limiting their upside. IBGYBG

LSWCHP's avatar

I had a 40 year career in real time embedded software development in all roles from junior engineer up to department head. I spent 30 years working for a defence company, and when I retired I can guarantee that nobody would ever be able to replace my institutional knowledge. The company goes on, but everything is more difficult, particularly support of legacy systems.

Everything is exactly as you described. Meanwhile, the universities are full of foreign students who pay lots of money, but are...how can I put this...not premium human capital.

I'm really going to be annoyed once the infrastructure (electricity and water supply and sewage removall that underlies western civilisation starts to fail, and like Stinger rockets, can't be repaired or replaced due to loss of institutional knowledge.

Shawn Willden's avatar

If the foreign students aren't premium human capital, you need to get better foreign students. The US has 4% of the world's population. If you're searching for the world's best and brightest, most of them will be from outside the US.

Jessica Luo's avatar

Foreign students not "premium human capital"?!

Enon's avatar

The problem is 100% managerialism. Not the particular managers, though they suck, but the whole idea that managers have a right to run things, that managing is a fungible skill that can be applied to managing anything, that managers' lines-of-code and all their other metrics and "work" are not just counterproductive idiocy. Managers, like AI, have no world-model, no map of reality, just lots of disconnected factoids which they heard in school or on media and repeat mindlessly. They also think economists know things, and that technology is an undifferentiated magic fluid. B-Ark cargo cultists. They're not sentient.

But one thing they are sure about is that they know how to rate people. Of course they're even more wrong than usual, managers instinctively hate their betters and always weed them out, it makes managers feel practically godlike to screw over thousands of people, mostly better than they are, when hiring for a single job. These supposed experts at dealing with people can't seem to figure out how many and how bitter enemies they make each time they do this. Perhaps they imagine that the applicants are lower status than their exalted managerial selves, so can't be angry? Or something? I think that not really being people, but rather badly programmed status-seeking automata, managers have no theory of mind.

You wrote: "Technical expertise alone isn’t enough anymore. You need people who can take ownership, communicate tradeoffs, push back on bad suggestions from a machine that sounds very confident. Leadership qualities. Our last hiring round tells you how rare that is: 2,253 candidates, 2,069 disqualified, 4 hired. A 0.18% conversion rate."

You have no ability to judge technical expertise or any of the other things you say you judged. Your hiring process is utterly retarded. I guarantee all the best applicants were disqualified, and the best people, that you should have recruited (meaning sought out and made offers to without application process) did not apply at all, because it's not worth going through a time-consuming humiliation ritual for a 1 in 600 chance of getting further fucked around by morons for a shit wage.

whisperer's avatar

Hi Denis,

If western capitalist economies insist on killing the pipeline (purely out of greed and gluttony) for the production of skilled personnel, then what can we do but let it burn?

I am very much looking forward to shortages of people who know what they are doing, which if you are right will be happening rather soon. It will put us in much stronger negotiating positions.

The problem if of course people that are trying to break into their respective industries, however such shortage will probably benefit them as well.

As for wether they can compensate with AI, I doubt it. Look at the recent Mythos release. The AI industry is non stop yapping about replacing all white collar labor, but the best they can do is marginally better and substantially more expensive tool for cyber and a lot of propaganda/hype/fud, if even that.

From what I am reading from Davi Ottenheimer, it is a lot less of an improvement on previous capabilities than what Anthropic wants us to believe.

Anyway, stay safe.

Denis Stetskov's avatar

When the defense supply chain failed, it didn't create a nice market for skilled workers. It created a crisis that affected everyone on the receiving end. If the software talent gap hits hard enough, it won't be a negotiating advantage. It'll be an economic problem that drags quality of life down across the board. Globalized economy means nobody sits this one out comfortably. That said, you're not wrong. Let it burn.

David's avatar

The thing that must be remembered about workers bring replaced by AI, is that if AI is better is not a consideration. If AI can do a passable enough job (even if much worse) and do it cheaper then most executive suites will jump onboard.

Of course if AI systems were charged out at a rate that would actuslly break even, never mind make a profit, they would be far more expensive.

Denis Stetskov's avatar

You're probably right, but we don't know the real cost of AI yet. Prices for newer models are creeping up, and there's already talk about restructuring subscription tiers. The honeymoon pricing that made AI look cheaper than people may not last.

David's avatar

We do know roughly how much money they are burning through. And we know how little they are earning. So we can make a good guess at what would need to be charged for them to make profit. And it is a lot.

Peter Papineau's avatar

And soon it will be reading and writing….

Jessica Luo's avatar

That's what I was thinking too. And that is even scarier!

z3n1th's avatar

One tension that exists in current SOTA alignment efforts appears between the helpful part of HHH (helpful, honest, harmless) and everything else that has a reward signal. Optimizing for the helpful "assistant" persona forgoes something that could be extremely important i.e., ensuring an environment where humans gain a thorough understanding of the stack/architecture they're working with and can contribute meaningfully in said domain. Rewarding models for achieving genuine dissemination of knowledge seems to be completely lacking by default in the current regime (the end-user being pre-disposed to such inquiries and willing to invest their time into utilizing the model for their own uplift of knowledge and understanding being a clear exception). I suspect there is a general dislike for "teachy" personas - thus models optimized along that line would be in direct conflict with existing market incentives see e.g., the backlash against GPT-4o deprecation. If frontier labs adopt training strategies optimized for teaching I think it is reasonable that a similar backlash might occur and cause reversion(s). Chain of thought is hidden by default in many clients marketed towards coding/SWE unless manually triggered/inspected, if the end-user gains an advantage by skipping permissions and just asking for the fastest route to accomplish X then examining CoT is probably not high on their priority list and neither is security. A good question to ask might be - which future approaches should be considered reliably sound given that the incentives driving the companies building these systems may be inherently corrupt and consequently any future models produced will likely be optimized for outcomes that look genuinely good but end up being terrible in the limit.

HowardNYC's avatar

hi... I'm an 64YO old fart

so old I can recall this horrid nonsense as occurring in the early 1980s, late 1990s, mid-2010s and here we are...

happens at about 15Y intervals

my personal favorite?

date and time calculations and the storage of D/T values

convincing everyone to have a corporation wide set of standards being a new war fought every time I landed new consulting contract at another corporate site...

please read this aloud: 01-09-16... 2016? 2009? 2001? JAN or SEP or illegal month code?

just for giggles... will everyone tell me the precise meaning of 12:00:00AM versus 12:00:00PM?

should 23:59 rollover to 24:00? or 00:00? or 00:01?

does everyone agree about definition of leap year / leap day?

how to handle international dateline?

number of timezones?

schedule of daylight savings time changeovers from 1980 to 2029?

{ time out for kitten videos }

Denis Stetskov's avatar

15 year cycles is an interesting observation. And datetime is the perfect example of Fogbank-type knowledge. Looks simple, every senior engineer has war stories, and AI will confidently give you the wrong answer for edge cases that took decades of bugs to discover.

HowardNYC's avatar

80/20 rule... really 80/20 rules

ditto for 90/10 rule being multiples as well

Erik Olson's avatar

That's not a sufficiently complex example. I recognize the horrible mess of date-time objects (I worked on Y2K but I gotta tell you the Euro switchover was a much bigger deal.)

For a while now, the answer to date-time is ISO 8601, Olson TZ database (last I checked. No relation) And removing all the bloody workarounds.

HowardNYC's avatar

ah! such a lovely dream!

George Bellarious's avatar

When it processes your docs, how do you know it hasn't injected myriad errors, aka "hallucinations"? I find LLMs don't even summarize articles correctly half the time.

Denis Stetskov's avatar

That’s not how it works. We don’t feed documents into an LLM and hope for the best. Text gets split into chunks, chunks get transformed into vectors. User input becomes a vector too. The system finds the most similar vector and grabs its neighbors for context. That context goes to the LLM. It’s mostly math. The LLM has much less room to hallucinate when it’s answering from a specific chunk than when it’s generating from memory.​​​​​​​​​​​​​​​​

john f.'s avatar

Regarding the impurities in Fogbank that made it work, you wrote: "That fact existed nowhere in any document. Only the workers who made the original batch knew it."

This is false. The workers who made the original batch didn't know it either. This fact is well established. You also wrote that they discovered in 2000 that they couldn't reproduce it, but this exaggerates how difficult it was to solve, because it wasn't until 2007 that they had a problem making the material. You cite "$69 million additional" for the reverse engineering but this is also incorrect; this was the total cost overrun of the project, not all of which was due to the costs of reverse engineering.

It's hard to trust an article with this many mistakes.

Denis Stetskov's avatar

You're right. The original workers didn't know about the impurity either. Los Alamos described it as an unknowing dependency in the process. I got that detail wrong. Fixing it. Honestly it makes the argument scarier, not weaker: even the people who built it didn't fully understand why it worked. Thanks for catching this. Fixed.

1123581321's avatar

Here's the really scary bit: we don't even have a good handle of how big this problem is, because it's only exposed when the chain is interrupted. In semiconductor manufacturing, it's a well-known problem with transferring technology between fabs - even exactly same equipment and exactly same recipe does not guarantee the same result. Sometimes it's as bad as "it doesn't even work", for example, the yield drops below an acceptable level so wafers have to be scrapped.

K Sokol's avatar

As Mr. Taleb once aptly wrote - if effective mangers could, they would remove extra lung, extra kidney extra anything from humans body in sake of cost optimiation.

Richard Fulmer's avatar

If we’ve forgotten how to make things, then how do you explain the following:

Total industrial production by year:

https://fred.stlouisfed.org/series/INDPRO/

Total manufacturing production by year:

https://fred.stlouisfed.org/series/OUTMS

Industrial output:

https://fred.stlouisfed.org/series/IPMAN

Total U.S. Industrial Capacity

https://fred.stlouisfed.org/series/CAPB50001S

Military procurement has been a nightmare for decades, but that’s a problem with government bureaucracy, not American industrial capacity.

Denis Stetskov's avatar

Aggregate industrial production going up and being unable to make specific things aren't contradictory. The US makes more stuff than ever. It just can't make Stingers, Fogbank, or 155mm shells at the rate it needs. That's the point. Specific capabilities and the knowledge behind them atrophied while total output grew. The FRED data is real. So are the rebuilding timelines. Same with code. More code is being written today than five years ago. Does that mean there are more engineers who can debug it?

Richard Fulmer's avatar

Again, the problems with making munitions is government bureaucracy.

Shawn Willden's avatar

It's interesting that you asked specifically about whether engineers can debug the code... in fact debugging is an area in which the LLMs are incredibly good. They do have a tendency to get stuck in local minima and need a human to nudge them to step back, but their debugging ability is superhuman. I've been writing and debugging code for nearly 50 years and I'm quite good at it... but while I still write some amount of code, I never try to debug when problems crop up, I always have Claude debug, then explain the root cause to me.

Denis Stetskov's avatar

Bureaucracy and knowledge loss aren't competing explanations. Bureaucracy created the conditions, knowledge loss is the consequence. Your Challenger Force fixes the feedback loop. The question is whether the talent to staff it still exists after thirty years of pipeline erosion. Interesting proposal.

1123581321's avatar

And then there's a weird relationship between the fragility of "making things" and the utter ignorance of the AI prognosticators of this. A perfect example: https://www.planned-obsolescence.org/p/six-milestones-for-ai-automation forecasts that in 2032 - yes, in six years! "AI" will take over the most complex production chain ever invented. I even asked, "do you mean, fully automated?", and yes, she meant "fully automated", because "AI2027". No, seriously. The lack of understanding of complexities of making things is astonishing.

We just don't have a perfect way to preserve critical details except by mentorship and on-the-job experience, be it in a bakery or in a semiconductor fab. No amount of written documentation replaces an engineer with 30+ years in the field having seen things go pear-shaped.

Denis Stetskov's avatar

Amodei said two days ago that coding is going away first, then all of software engineering. That’s the CEO of the company whose model I use daily. Meanwhile his careers page lists 146 open software engineering positio

1123581321's avatar

Yeah, it's almost like he believes that "generates code" = "software engineer", a view clearly not shared in the trenches.

RegieRoger's avatar

i would say the loss of important corporate knowledge is not a bug but a feature. people would rather come to the rescue and solve an avoidable problem rather than manage it without any recognition

Denis Stetskov's avatar

Hard to argue. Firefighting gets promoted, prevention gets ignored. Same incentive structure that kept defense budgets low for thirty years. Nobody gets credit for "nothing went wrong."