Product Fundamentals is a podcast dedicated to spreading the core knowledge software product people need in order to succeed. Season 1 is structured as a history of how we make software.

As the dot-com bubble came crashing down, a group of "organizational anarchists" got together at a ski resort and created the Agile Manifesto, finally providing a common banner for incremental software people. One of the most successful offshoots of Agile would be Lean Software, which took its foundations from Toyota's lean manufacturing system and brought it to software, setting the stage for the startup wave that was to soon follow.

The audio is embedded below, and the episode transcript follows.

You can also find this episode of the Product Fundamentals podcast on the show website and through all the usual podcast services.

Transcript

Hello friends, and welcome back to the Product Fundamentals podcast, episode 6: Agile & Lean. 

In this season, we are tracking the evolution of how we came to make software in the weird way we do, from the earliest origins of our methods, through to today.

Last episode, we discussed how the changing technological and cultural environment of the 1990s combined with growing discontent over the failures of the Waterfall methodology, leading to the emergence of new methods like Scrum and Extreme Programming.

This time, we’ll discuss how those Waterfall alternatives evolved into their most lasting articulations: Agile and Lean.

Before we get into the history of these two concepts, it’s worth re-acknowledging what I hinted to back in the preamble to this whole season: our modern way of work is pretty muddy when it comes to ideas like Agile and Lean. Are Agile or Lean specific methodologies with specific practices and rules, or are they catch-all terms for “do good stuff”? Are they a positive program for something, or simply a negative statement against what came before? Are they the same as one another, or are they different?

The answer to all of these is… it depends on who you ask, and when you ask them.

But from a historical perspective, we can tell a pretty clear narrative, and then we can circle back to making sense of it all at the end. So, let’s get into the history.

On March 10 of the year 2000, the Nasdaq stock index, which contains the stocks of most publicly-traded technology companies, hit its all-time high of 5132 points. That reflected just over 100% growth the preceding year, and 360% growth since 1996, just four years earlier. This moment in March 2000 was, of course, the financial peak of the dot-com bubble.

The peak was short-lived. A year later, in March 2001, the NASDAQ Composite index was already down more than 50% from its peak, and was on a steady continuing decline that would see the index lose another 50%until it hit bottom in October 2002. In total, the NASDAQ index would lose 78% from peak to trough, giving up all its gains since late 1996.

As I record this in mid-2023, there’s plenty of economic anxiety in the software industry, as interest rates rise, generative AI asks challenging questions, and layoffs seem ubiquitous. But we’re not even in the same universe of pain as the dot-com bubble. TheGlobe.com went from one of the largest IPO pops ever to de-listing in just over three years. Webvan went from IPO to bankruptcy in 18 months, vaporizing 800 million dollars of venture capital in the process. Pets.com took the cake, going from IPO to bankruptcy in just eight months. But the crash didn’t just take down obviously silly businesses – it also crushed solid businesses, some of which survived by the skin of their teeth, such as Amazon and Priceline.

The Agile Manifesto

It was in this environment, where the sky was truly falling, that supporters of Extreme Programming had held a number of meetings and online conversations, which culminated in the decision to organize a summit of sorts with practitioners of various anti-Waterfall methods of building software. Clearly, something was written in the state of software, and it was time for a change.

Invitations went out to inventors and advocates of XP, Scrum, the Ruby programming language, various design approaches for using object-oriented programming, and other IID methodologies.

In February 2001, seventeen self-described “organizational anarchists” met at Snowbird ski resort outside Salt Lake City, Utah, to synthesize their perspectives into a shared statement about how we should make software.

That group of 17 agreed to a shared set of values for how they wanted to work. In naming themselves, they wanted to mark a clear split from the “heavy” processes that dominated the software industry, but they rejected the word “Light” as sounding too trivial. So, instead, they settled, of course, on “Agile.”

The outputs of the conference were a new non-profit organization (called the Agile Alliance), a set of published principles, and a very short document that they termed The Agile Manifesto.

Just 68 words long, here is the complete Agile Manifesto:  

“We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:

* Individuals and interactions over processes and tools
* Working software over comprehensive documentation
* Customer collaboration over contract negotiation
* Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.”

And that’s it.

The 12 principles add a bit more concrete detail. I won’t read them verbatim, but to summarize, they say: deliver working software to the customer early and often, embrace changing requirements, tech and business should work together every day, management should support and empower self-organizing teams, face-to-face conversations are the best form of communication, teams should work at a sustainable pace that can continue indefinitely, technical excellence and good design deserve continuous attention, simplicity is essential, and teams should have regular retrospectives to figure out how to improve. That’s it. Not a very tight blueprint.

In retrospect, Agile – or something like it – was probably inevitable. We talked about its precursors last week, as the weaknesses of highly-documented Waterfall grew more evident, and new ideas took root. But it’s also easy to say it was inevitable, because Agile is mostly just a protest banner. As we heard, the Manifesto itself is oppositional, and the attached principles are broad. The Agile of 2001 was a pretty empty vessel that could be filled with all manner of different contents.

In the years immediately following the Manifesto, 15 of the 17 attendees of the Agile summit wrote books defining their personal take on how to do Agile. They weren’t the only ones – it seems that everyone who was anyone in early 2000s software hard to write a book about their approach to Agile. Don’t worry – I’m not about to run through a dozen books defining Agile. The broad strokes are generally the same, and the biggest constants are pulled straight from Scrum and XP: short iterations called sprints, including the customer in the process, relatively small autonomous teams – you get the picture.

While there was nominally a single organization, called the Agile Alliance, at the center of all this, whatever authority they might have had to set an agenda quickly got lost. With all these many voices, Agile rapidly became a broad banner rather than a specific practice. Its appeal was undeniable. From its inception in 2001, Agile has enjoyed a steady, inexorable climb in prominence. Google’s Ngrams tool shows the use of the word “Agile” in books began a long linear growth in prominence from the time of the summit through to the most recent data from 2019; web search Trends show similar continuing growth. Agile immediately captured the zeitgeist, and hasn’t let go.

Agile as cultural moment

Last episode, we touched on Extreme Programming as an early exemplar of the class consciousness of programmers, and the accompanying “coder as craftsman.” That notion only strengthened with the Agile Manifesto. Heck, just using the term “Manifesto,” with its radical political overtones — and then having the document be just 68 words – was as much an act of cultural expression as business strategy.

The independent, anti-corporate thread of software culture is clearly on display in the early days of Agile. Just a millimeter below the surface of the Agile Manifesto and its accompanying materials is a rejection of the suits who want to tell engineers how to work, and a desire from workers to claim greater control and ownership of what they build.

It’s certainly possible to take a cynical view of the Agile founders and its follow-ons. As mentioned, The Agile Manifesto’s authors were consultants and working senior engineers, most of whom went on to sell books and courses evangelizing their preferred flavor of the methodology. While they were challenging the prevailing way that software was made, it’s hardly novel to accuse large companies and government agencies of being slow and inefficient, especially after a financial crisis. Despite the radical flavor, the Snowbird 17 weren’t exactly sticking their necks out.

Still, the rapid adoption of Agile, and its clear staying power in the collective consciousness of software workers worldwide is evidence that it tapped into strong widespread sentiments in the industry.

Economic changes

With the dot-com crash and the resulting job losses also came an even more severe collapse in venture-capital funding. While modern Venture Capital has existed since the 1950s, the dot-com bubble had been its glory days. From less than $10 billion dollars of investment  across the US economy in 1995, VC reached its peak 12-month period, at the end of 1999 through the fall of 2000, when venture capitalists invested 126 billion dollars across all industries in the US. Then the  hammer dropped, and it dropped fast. The worst 12-month period of the crash covered the second half of 2002 into 2003, when just 15 billion dollars were deployed. That’s an 88% decline in VC funding across the US economy in less than three years. 

In inflation adjusted terms, American VC spending would not exceed its 2000 peak until the zero-interest rate pandemic bonanza of 2021.

But the underlying technological advancements and exponential growth were continuing apace. Moore’s Law, Kryder’s Law, and Dennard Scaling, were joined by the growth in Internet users and bandwidth. Nielsen’s law of Internet Bandwidth, coined in 1998 by Danish web usability consultant Jakob Nielsen, observed that the bandwidth of a high-end consumer Internet connection would increase by 50% per year; the rule has held empirically true since 1983 and remains in effect as of 2023.

Moreover, in the early 2000's, the share of the world population that was using the Internet on a regular basis was doubling roughly every three or four years. 

Thus despite the dot-com collapse, the market opportunity for Internet software remained enormous. But given financial conditions, pursuing those market opportunities required methods that could succeed despite tight resource scarcity.

To meet this challenge, some American software professionals turned to the model that had emerged from a very different time of scarcity, in a very different place. They looked to Lean Manufacturing.

Lean manufacturing

The Lean Manufacturing Production System was developed in the 1950s and 1960s by Japanese manufacturers – especially the Toyota car company. I think to really make sense of Lean, we need to understand the economic and social context from which it emerged.

In post-war Japan, capital was scarce and carefully guarded. It is hard to overstate, let alone to really comprehend, how completely devastated Japan was by three years of Allied bombing. Forty percent of the urban area of Japan’s cities had been destroyed. Overall industrial production capacity fell by 50% in just the last year of the war, with heavy manufacturing for war-relevant goods like airplanes crushed even further. The Allied bombing of Japan had rendered some 8.5 million people – more than 11% of Japan’s population – homeless. Japan’s small and mountainous home islands are poor in many natural resources, and overwhelming damage to infrastructure, like bridges, ports, and railways made it hard to import or mobilize whatever materials they could muster. The bombing had killed perhaps half a million people – many of them factory workers – and another 2 million prime-age men had been lost to battlefield deaths.

As you might imagine, after the war, financial systems were slow to recover. The Tokyo Stock Exchange didn’t reopen until 1949.

In this environment, it was simply impossible for shaky post-war Japanese firms to finance and construct massive production line factories, to carry large inventories of spare components, or to tolerate significant defect rates in their products.

Rather, Japanese firms in the 1950s had to ruthlessly optimize for capital efficiency. This drive for efficiency drew Japanese firms toward the Scientific Management of Frederick Taylor and the like, which we discussed briefly in part 3 of this series. This is the approach to manufacturing management that called for analyzing and optimizing every step of an industrial process until it was as efficient as possible. An updated set of ideas about improving production had also come in with the American occupation, in the form of American management consultants William Edwards Deming and Joseph Juran. 

We touched on Deming a bit in episode 2 of this series, while discussing the industrial precursors of iterative and incremental development. Working separately, Deming and Juran had a large influence on Japanese industrialists in the 1950s through their lecture tours and consultations. Deming advocated for high quality product design, for using statistical analysis to analyze production methods and improve quality, and for exporting to foreign markets to drive growth. 

Joseph Juran focused on quality management, meaning the set of practices companies should adopt in order to maximize product quality. He advocated for seeing a product’s usefulness to customers as the key measure of quality, and for seeing poor quality as a cost to the business. Finally, he put particular emphasis on the need for a top-to-bottom culture of valuing quality, from senior management through to the factory floor, reinforced through training at all levels and empowerment of workers at all levels to improve quality.

This somewhat softer and more inclusive evolution of scientific management was consistent with the labor situation facing managers in post-war Japan. The Allied Occupation government, led by American New Deal Democrats, had legalized and encouraged labor unions, seeing them as a democratizing force. The largely US-written constitution of Japan enshrines the right to unionize, and union membership peaked at nearly 56% in 1949.

While capital was scarce, so was skilled labor, and that labor had leverage. To gloss over a complicated mix of historical developments and cultural background, what eventually emerged in the 1950s and 1960s in Japan was a compromise system in which Japanese firms all-but-guaranteed lifetime employment to their workers, while labor unions often narrowed to being company-specific, meaning that a union would represent just the workers at a specific firm. By the late 1960s, many of these “enterprise unions” were in the process of transitioning from strikes and agitation toward more cooperation and collaboration between labor leaders and management.

Taken together, the economic realities of post-war Japan, new ideas about industrial production, and the system of empowered lifetime employees planted the seeds for a big change in manufacturing.

Industrial engineer Taiichi Ohno and executive Eiji Toyoda, both of the Toyota Motor Corporation, spent nearly 30 years refining the car company’s production methods to succeed in this milieu. The resulting Toyota Production System, or TPS, is a just-in-time system, meaning that components should be built or acquired only when needed, and only in the quantity needed. There should be very little spare inventory at any point in the production process.

TPS places a nearly-fanatical emphasis on the reduction of all kinds of waste. Overproduction is a waste, unproductive time is a waste, processing is a waste, transportation is a waste, defects are a waste, underutilized workers are a waste, and so on.

In the Toyota Production System, teams of workers with different specializations generally own several steps of the production process, and are empowered – and expected – to improve efficiency and reduce waste in their domain. In fact, any worker on a Toyota production line is able to, and expected to, hit a button to stop the line if they spot a defect in their own work or the materials coming from upstream of them. When such an issue is spotted, the process that created the defect is reviewed, and an improvement is proposed to prevent future defects. Because the worker has a near-guarantee of lifetime employment, there is little risk of losing one’s job for causing or noticing a problem; for the same reason, the worker also feels greater investment in the success of their team, product line, and company than workers in more precarious jobs in other countries.

While we’re in the neighborhood, It’s worth noting that on the design and development side, Toyota uses matrix management, which we described back in episode 2. A chief engineer serves as the project leader, pulling in experts from all of the various functional specialties for the duration of the project. The specialist engineers develop deep knowledge and experience in their area of expertise as they continue to reside in one functional group, while contributing to many different projects over a career.

Other important characteristics of TPS include a preference for using proven technologies, rather than cutting edge ones – this reduces risk and waste. Employee development through training and support is key, because of the long-term relationship between workers and the company. Decisions are made slowly, through consensus-building, but the resulting decision is then implemented quickly and decisively. Constant learning, self-reflection, and continual improvement (kaizen) are essential.

Another important element of TPS is the collaboration between a manufacturer like Toyota and its many suppliers. Toyota was, no doubt, a demanding customer, and set high expectations for its component manufacturers. But Toyota was also an investor in and collaborator with its suppliers, sharing its production method and working together with certain suppliers on some projects. The resulting diffusion of knowledge and technology throughout the ecosystem of suppliers helped those companies grow and succeed relative to their competitors outside the Toyota web. Being in the ecosystem paid dividends to Japanese suppliers that the more extractive relationships between American auto manufacturers and their suppliers did not.

The Toyota Production System also gave us the kanban, commonly adopted for tracking progress in software today, which in the manufacturing context was used to track inventory levels of different components at different stages of production. TPS  is a pull-based system, meaning each step of the process should be sending a “pull signal” to the steps upstream of it when it runs low on some input, triggering more production upstream. If the supply of components either runs out or accumulates a large backlog at any point in the system, then there must be an inefficiency that should be addressed.

Through independent development and explicit copying, the Toyota Production System quickly diffused throughout the auto industry and then into adjacent manufacturing subsectors in Japan. Though the label “Lean” wouldn’t be coined until 1988, the difference between Japanese and American production methods had become clear much earlier. By the time Japanese car manufacturers started really entering the US market in the late 1960s, the Japanese firms had trained themselves to build higher quality products at much lower cost than the American giants could.

The difference became especially stark when the 1973 Oil crisis hit, driving up the price of oil four-fold, and thus increasing demand for smaller cars. The Japanese firms were already producing smaller vehicles, giving them an advantage, but they also had a workforce, design process, and production system well-equipped to adapt to a changing ecosystem. In contrast, the oligopoly of American auto manufacturers had huge inventories of parts and cars to clear, slow design processes, an industry-wide union with little interest in the success of any one company, and a cultural resistance to changing plans because of customer desires. In just over a decade, Japanese manufacturers went from less than 1% of the US auto market to about 20% in 1980 and 30% in 1990. 

In order to get closer to their customers and avoid trade restrictions, Japanese firms opened manufacturing plants in the United States, which helped further introduce Americans to how lean manufacturing works, and which served to illustrate that lean manufacturing could work in an American context. Through joint ventures with their Japanese rivals, as well as through home-grown iterations on their production methods, American auto makers finally adopted lean manufacturing practices in the 1980's and 1990's, with Ford perhaps the most successful. 

With these Japanese challengers successfully disrupting and spurring change in such an established pillar of the American economy, it was natural that software practitioners of the period, dissatisfied with the sclerotic methods of their industry, tightly constrained on resources, and eager to chase a booming market, would find inspiration in Lean manufacturing.

Lean software development

The ur-text for bringing Lean to software is, appropriately enough, titled Lean Software Development: An Agile Toolkit, published in 2003 by Mary Poppendieck and her husband Tom. 

Mary had been a process engineer and IT manager at a factory, and then a product owner at 3M. From this industrial experience, Mary learned about the Toyota Production System. Tom had been a physics professor and software engineer turned consultant. One gets the sense from the book and other sources that Mary was the primary author and driving force behind Lean Software Development, but since the book credits them both, I’ll refer to the Poppendeicks collectively throughout this episode.

The Poppendiecks had not been at the Snowbird Summit. But they were among the fast-followers of the Agile Manifesto, positioning “Lean Software” as a more specific runbook that fits under the broad Agile umbrella.

Lean software development has seven “principles,” which the authors say represent a distillation of Lean manufacturing techniques. They are:

• Eliminate waste
• Amplify learning
• Decide as late as possible
• Deliver as fast as possible
• Empower the team
• Build integrity in 
• See the whole (w-h-o-l-e).

By this point, I hope we’re all comfortable with the basics of modern IID, so I’ll touch only briefly on some of these principles. Others bear a bit more attention.

Eliminate Waste is the shibboleth of Lean Manufacturing, so of course it makes an appearance in Lean Software. To quote from the Poppendeicks, 

“Waste is anything that does not add value to a product, value as perceived by the customer. In lean thinking, the concept of waste is a high hurdle. If a component is sitting on a shelf gathering dust, that is waste. If a development cycle has collected requirements in a book gathering dust, that is waste. If a manufacturing plant makes more stuff than is immediately needed, that is waste. If developers code more features than are immediately needed, that is waste.  The ideal is to find out what a customer wants, and then make or develop it and deliver exactly what they want, virtually immediately. Whatever gets in the way of rapidly satisfying a customer need is waste.”

The nature of the relationship with the customer over time is a key thread to consider. In the early days of commercial software, the customer was a giant entity – often the government – commissioning something entirely new from scientists operating at the edge of what’s possible. By the time of Waterfall’s ossification into DoD standards, the relationship with the customer had become legalistic, defined by exhaustive contracts with thousands of explicit requirements. For Scrum and XP, the customer was a partner, who should be physically in the room and engaged throughout the software development process. Now, as the industry processes the scars of the dot-com crash, the customer is becoming the target of obsessive service. Everything that does not make the customer happy is waste. This mentality will play a critical role in the coming startup wave of the 2000s, which we will discuss next episode.

Amplify Learning is about using rapid iteration to test ideas and learn things as fast as possible. The Poppendiecks claim that in a scientific sense, the maximum amount of new information is gained when the probability that a hypothesis fails is 50%. Thus, when operating on a novel problem – in unfamiliar territory, as it were – we need to learn quickly while still delivering value, which means we need to make rapid small iterations that allow us to go in incorrect directions, learn, and change.

In order for the learning to be valuable, we can’t have a tightly determined sense of the destination. Once we’re anchored on an outcome, feedback becomes dangerous, because it can threaten the plan.   

Deciding as late as possible means rejecting the desire for settled decisions before any action can begin. Just as Takeuchi and Nonaka described in The New New Product Development Game in our last episode, phases of a project run concurrently rather than sequentially. Each worker, team, and project must preserve enough flexibility in their plans to accommodate changing requirements. We design for change, avoid making commitments to specific features until the last minute, and then develop the capability to respond quickly when there really are pressing needs to be satisfied. Thus we avoid “deciding” our way into deadends and traps.

Deciding as late as possible leads naturally to the next principle: Deliver as fast as possible. The faster we deliver, the later we can decide, and the more relevant the product we deliver can be, because it’s based on the most up-to-date information.

This idea leads to the “just-in-time” and “pull” systems of production, just like TPS. It is better for engineers to be pulling just-barely-ready items from earlier steps in the process, rather than having a lengthy backlog accumulate upstream of them. Thus, the use of the kanban – the visualization shows work in progress at various stages of development, highlighting where there are surpluses or bottlenecks in production.

Empowering the team stands out from the other lean principles because it’s incumbent on management rather than the individual contributors to make it happen. In Toyota, individual team members have the career safety and the autonomy to improve how they work. Lean software is built around the idea that, contrary to Frederick Taylor, workers want to do well and earn respect through excellence. They don’t need to be told how to perform their jobs and then relentlessly monitored; they need to be given the space, resources, and incentives to discover how to do the job well themselves.

In applying this principle to software management, the Poppendeicks echo The New New Product Development Game: management must give teams a clear, compelling, and achievable purpose, provide access to customers, give the team latitude to make its own plans and commitments, supply resources, and shield the team from distractions. Teams also need recognition for success, to reinforce the intrinsic motivation for excellence that the whole system is predicated on. While they don’t dwell on it, the Poppendeicks to also express passing skepticism with OKRs and other management tools as more likely to feed job dissatisfaction rather than motivation. They write, 

“We believe that the critical factor in motivation is not measurement, but empowerment: moving decisions to the lowest possible level in an organization while developing the capacity of those people to make decisions wisely.”

Build Integrity In refers to two linked concepts, which had been previously described in research into how some car companies delivered consistently superior products than their competitors. These concepts were perceived integrity, and conceptual integrity.

Quoting the Poppendeicks, 

“Perceived integrity means that the totality of the product achieves a balance of function, usability, reliability, and economy that delights customers. Conceptual integrity means that the system’s central concepts work together as a smooth, cohesive whole.”

You may remember from Episode 4 that Fred Brooks, in The Mythical Man-Month, thought that what he called “conceptual unity” was critical to software success, and that in order to achieve conceptual unity, projects needed to be anchored in the vision of a singular wise chief architect.

While certainly less hierarchical than Brooks’ approach, the Poppendeicks’ lean software still has a concept of the Master Developer, who is the respected leader of a project. But they soften a project’s reliance on this central figure by increasing developer-customer contact, having customers test iterations, and by the use of model-driven design, in which the customers and developers work together to define the set of concepts in the system (which is to say, the classes of objects in an object-oriented program) and how they interact. Building a shared conceptual model used by both customers and developers should increase the integrity of the result.

“Building integrity in” also becomes the umbrella for practices like automated testing and refactoring. Automated testing ensures that the system remains reliable, maintaining its perceived integrity. Regular refactoring reduces waste by preventing the build-up of brittle failure-prone systems and increases conceptual integrity by ensuring consistent principles are applied across the program.

The final principle of lean software is See the Whole. The point here is captured in the maxim, “Beware of the temptation to optimize parts at the expense of the whole.”

In the weeds of a software product, this has a literal sense which blends into the previous topic of Integrity. All the part should work well together. But seeing the whole also applies to many areas of measurement. In performance management, the Poppendeicks express skepticism about rubrics that pick out a few areas to measure and rely on as indicators of holistic performance. In evaluating failures, they caution against attributing them to a narrow cause or to an individual rather than to systemic problems. Resist the temptation to focus on easily-seen and measured symptoms, rather than on elusive root causes.

One last bit to pull out of the book is a good reminder of the times it is being written in: about half of the “see the whole” chapter are about how to structure contracts between customers ordering software and the firm building it on demand. 

Since Agile and Lean are still such common concepts in modern software development, it’s easy to forget that even these concepts are kind of historical now. In 2003, we still weren’t in the Software as a Service era. The only specific Internet software product that I noticed getting a mention in the book was Google’s basic search engine, which only got mentioned in a brief aside. The methodologies that are such touchstones today are, themselves, products of a very different time. 

Lean software and Agile

So, with all that said, what do we make of Lean Software?

First, let's touch on its relationship with Agile.

As one of the first methodologies to be articulated after the Agile Manifesto, Lean software development became closely associated with and representative of Agile overall.

This wasn’t some sneaky co-opting; the forward to the Poppendeicks’ book was from the originator of Extreme Programming, Ken Schwaber. And with Agile being a loosely-defined Big Tent, there is room for lots of specific methodologies to call themselves Agile. In that sense, Lean is just a specific flavor of Agile.

But there are also some tensions and differences in emphasis. In Lean Software Development, many of the prescriptions of XP and Scrum are gone or much-softened. XP was all about taking best practices to their extreme; one can readily imagine those extremes being leading to Waste. While Lean is iterative, the Poppendeicks break from Scrum by being ambivalent about how long the iteration should be, what meetings and artifacts are needed. XP and Scrum were highly prescriptive and operated at a very concrete level, and the Agile Manifesto was highly abstract and flexible, Lean is somewhere in between on the abstraction ladder. Lean is a form of Agile, but one can certainly understand how other Agile folks would chafe against it. 

Lean manufacturing and software development

Of course, after running through the history of lean manufacturing and Toyota, we can’t close without talking about Lean Software’s relationship with its manufacturing namesake.

My perhaps controversial claim is… they’re really not that related.

Many of the differences have to do with scale. First, even lean manufacturing has an enormous up-front capital cost compared to software development. Even in the days before cloud infrastructure providers, the cost of a server, a few workstations, and a converted warehouse working space for a handful of software engineers paled in comparison to the cost of the leanest automobile production line. 

The nature of the product is very different. Car companies make significant investments to design a vehicle, and then manufacture many thousands of identical products. Even before software as a service, this was a poor parallel for software. The design and the production of software are much more nearly the same process than the design and manufacturing of physical goods. The cost of reproducing existing software is nearly zero; there’s no concept of producing the next unit of existing software more efficiently than the last unit.

In the 2023 world of constantly-updated online software, the parallel is even more strained. There are no fixed editions of most software; there is a single, continuously-evolving product. The only manufacturing process that we’ve talked about that mirrors that dynamic is the development of the X-15 aircraft in the 1950s from back in Episode 2. That was a small team of experts operating at the cutting edge of technology; that’s not at all like lean manufacturing, with its efficient scaled-up production of many identical units using well-established materials and technologies.

Lean software methods also don’t often create the networks of collaborating suppliers that lean manufacturing does. Most software companies practicing lean development don’t have the size, the clout, or the dependency on external suppliers to build a network of interlocked firms that collaborate and share ideas. Each company is an island, with informal social bonds and the poaching of talent between firms driving any cross-pollination of ideas, rather than structured programs.

Most importantly to my mind, and most often overlooked, is the labor structure. Having worked in software in both Silicon Valley and in Tokyo, I can tell you from personal experience that the labor protections of working in Japan really do change your relationship with your company and your work. For workers in more traditional industries like manufacturing, especially during the ascent years of the Toyota Production Method, the expectation of loyalty went both ways: companies couldn’t fire workers, and it was also difficult for workers to change jobs between companies mid-career. As a result, workers from the production line through to management had a long-term commitment to their company, simultaneously as a source of prosperity and as a source of identity.

The story could not be more opposite for workers in software, especially in the United States. Unions functionally do not exist. Job tenures of two years or less are very common, with workers changing company in pursuit of higher pay. Companies are free to fire workers with impunity, and as I record this in 2023, recent events have shown that cash-flush massive tech companies can and will fire tens of thousands of workers at once without any clear necessity beyond a vague sense of shareholder value.

Lean manufacturing relies on workers having strong commitment to both their employers and their teams, on workers having the financial and psychological safety to raise problems, on workers having the confidence their input will be taken seriously.

None of that exists in any formalized sense in the software industry in the United States and in other countries with similar labor laws and norms.

Given the centrality of a particular labor relationship to the construction and success of the lean manufacturing system, I find the application of the Lean label to software rather disingenuous. There are plenty of good ideas in lean software, but personally, I really wish the Poppendeicks had picked another name. 

Wrapping up

Of course, whatever these differences from lean manufacturing, Lean Software and the wider Agile movement stood in clear contrast to the creaking Waterfall paradigm that they had rejected, and they gained near-total dominance in the minds of software professionals.

A perennial follower in matters of software by this point, even the US Defense Department would finally revise its procurement guidance to explicitly favor Agile contractors in the 2010s. I even came across a 2018 document from the Pentagon’s Defense Innovation Board, called “DIB Guide: Detecting Agile BS”, which includes this delightfully self-aware quote:

“Agile is a buzzword of software development, and so all DoD software development projects are, almost by default, now declared to be “agile.” The purpose of this document is to provide guidance to DoD program executives and acquisition professionals on how to detect software projects that are really using agile development versus those that are simply waterfall or spiral development in agile clothing.”

When even the DoD is offering tests to distinguish between real and fake Agile, you know that the industry has had its worldview changed.

That’s all for this time.

In our next episode, we’ll see these methods for designing and building software brought to a much wider scope, as the startup wave of the 2000s turns the whole business into an experiment, reinvigorating the software industry and bringing MVPs, OKRs, mobile, and more to our way of work.

As always, your comments and feedback on this episode are very welcome. You can find a transcript, links to sources, and ways to reach me on the show website at prodfund.com. 

And if you like this series, and you want to hear more, do me a favor and share it with someone you think would enjoy it too.

Thank you very much for listening.