Browse Books

From the Book: A large majority of software development projects still cost too much, take too long, and don't deliver what they promised at the beginning. That's if they actually deliver something at all. A significant number get cancelled before there's any useful software to hand over. Not all software development projects fail, but it certainly can seem that way. In 2000, one industry survey claimed that roughly half of software projects were over budget, late, or didn't deliver what they promised. Another roughly twenty-five percent were cancelled before delivering much of anything at all.1 These numbers were an improvement over the figures for the first survey in 1994, where the results were abysmal, but this is hardly cause for celebration. Almost three quarters of software projects still produce junk, late, for a lot of money, and at the expense of people involved. If you ask industry observers and corporate IT leaders why this is so, I'll bet you'll hear a standard response. They'll tell you projects with more executive sponsorship, smaller scope, greater user involvement, firmer requirements, better estimates, better programmers, and more solid project management are more likely to succeed. Projects that deviate from these things are proportionately more likely to fail. These are good things, but this recipe for success isn't very satisfying. It's difficult, if not impossible, to have all of these things on a single project at once. Most projects can't, and don't, but they have to create software anyway. Even more disturbing is that some projects have all these things, at least according to the checklist, and still fail. Why Why are business leaders willing to spendmillions of dollars each year on sub-par products from IT, or from services firms they hire How come they're willing to spend just as much money, and more, on advice from IT management gurus whose advice often doesn't work, or is impossible for real human beings to implement The Manufacturing Mindset Software development in most big companies, and even in many smaller ones, is based on a single, unquestioned metaphor: We manufacture software. The Industrial Revolution created a psychology for work and management. People who lived during that time, especially in the first two-thirds of the twentieth century, thought about most aspects of life in terms of industrial production. Business leaders did the same. This was natural. Software development for mission-critical business systems began to gain serious momentum in the 1950s, in the heyday of the Industrial Age. It isn't surprising that software developers of that time, and the business leaders who employed them, thought about software development like they thought about other kinds of "production." In fact, traditional explanations about why software projects fail, and traditional methods of avoiding failure, assume that software development is a control and optimization problem, much like factory production. These explanations have held sway for almost fifty years. How would you increase quality and efficiency in a factory You would make things predictable. You would break work down into very small steps, and have workers do the same thing all the time. You would make sure to engineer what you're producing to the last detail, then you would determine the optimal production steps and schedule. If you assume software development is like a factory, and software developers are like workers on an assembly line, these success factors make sense. Unfortunately, the assumptionoften unconsciousthat software development is an efficiency optimization problem has led IT organizations to a dead end. Bad Assumptions The manufacturing mindset is familiar and comfortable for people born in the twentieth century. It's reassuring for us to believe that we can extend that mindset to software development, but this belief rests on three bad assumptionsassumptions that are reasonable in a manufacturing context but flawed in a software development context. These are: That we can make useful, detailed, accurate long-term predictions That we can control people and events to make our predictions come true That our attempts to control will produce the results we want All of these assumptions turn out to be wrong most of the time. We can only make useful, detailed, accurate long-term predictions if changes we make have proportional effects, and if all the pieces of the problem are independent. If changes can have disproportionate effects, or if the pieces of the problem are interdependent, then accurate, detailed, long-term prediction becomes impossible. Suppose I tell you that the Design phase of a project will be done on a specific date. What if the design isn't done Then the Design phase isn't really done. Or what happens if the team encounters a problem in the Build phase or the Test phase, and they have to redesign a certain component completely Then not only is the Design phase not done when I said it would be, but it also isn't done when we thought it actually was. And the component won't look like my prediction said it would either. The bad news is that these things always happen for systems that do much of anything interesting. When we plan at the level of detail most people are used to, we're trying to predict specific details about the long-term future of something affected by a dizzying number of variables that relate to each other in complex ways we can't sort out completely ahead of time. Even if we could predict accurately, though, we wouldn't be able to control everything like we want to. Real-life problems defy control. There are too many variables interacting in ways too complicated for us to understand completely. In common language, this is known as the Law of Unintended Consequences. The slightest change, a seemingly innocuous little modification, can have disproportionate effects that we can't predict. We have very little hope of controlling much of anything really complex. This gives already overstressed managers intense heartburn, which leads them to obey what Tom DeMarco calls the First Law of Bad Management: If something isn't working, do more of it. When things seem out of control, most managers react understandablythey try to control more. When they do that, they unintentionally squeeze out the potential for novel discovery their teams need in order to solve problems they've never solved before, which is what software development is all about. Managers bent on control force their teams to follow all the checklists for their industry certifications (choose your flavor) right into the ground. The best control-oriented managers can hope for is that their teams will produce the software people thought they wanted when the project started, not the software people end up wanting after they learned along the way. That's not the result anybody wants. These three bad assumptionsthat accurate, detailed long-term prediction is possible; that control will make our predictions come true; and that control will give us the results we wantexplain why software projects based on the manufacturing mindset so often end up in the weeds. What fascinates me is that, at least at some level, we know this ahead of time. We know we can't predict the future very well, but we act as if we can, and we take our attempts very seriously. The reason is that we forget we're all human, and we've been conditioned to think in certain ways by our experiences, by what we've been taught, and by the times we live in. When managers think and behave according to their conditioned mindset, they unconsciously lock their teams into the status quo of failure, frustration, and disappointment. A New Way of Thinking The manufacturing mindset might have made sense back when software development was a simpler problem. We're past that point, and the analogy doesn't work anymore. Software development today is a completely different animal. It's a creative process, not entirely mechanistic. Software emerges as we create it, sort of like a painting. What managers need is a new way of thinking and behaving which recognizes that software development is a creative process, but which still allows them to act intentionally and to make realistic plans. I call this new way of thinking growing software. This book is about applying a principle that seems too simple to most people. It is that we should grow software, not build it. Software development is a messy, creative problem that is virtually impossible to predict or control in the way we think it ought to be. If that's true, then our traditional approaches to planning and running software development projects make very little sense. Instead, software development should look more like human beings feeling their way toward solutions they can't quite put their finger on until they see them. I'm sure your own experience tells you this is not how most organizations develop software. When human beings try to solve messy problems like software development, they often act like what complexity theorists call a complex adaptive system (CAS). These systems share three important characteristics. First, complex systems are based on networks of agents acting in parallel. In general, these agents can be any components in a system, such as cells in your body or amoeba on a slide. The really interesting systems, though, are the ones with human agents. Human agents are not always rational, they use strategy in their interactions, they have to operate on incomplete information, and they have previous experience to influence their current behavior. The programmers, managers, sponsors, and other people involved with software projects are the agents of the software development CAS. Second, the number of possible actions each agent can take at any point is huge. As a system gets more complex, the space of possibilities for what happens next is vast. The best human agents in such systems can do is to use rough rules of thumb, built into collective memory, or history based on feedback from trial and error. These skills become the predictive model agents use to decide what to do next, without any centralized control. Agents "explore" their way to workable solutions. This is what the programmers, managers, and other people involved with software projects do, even when managers try to force things to go according to plan. Third, these imperfectly intelligent agents exploring their way to workable solutions are subject to conflicting constraints. They can't simply do whatever they want. Most of the time, they have to try to make many different stakeholders happy. In real life, it's virtually impossible to make everybody happy at the same time. The best agents can do is pick their battles and do the best they can. Software development projects are like this. Programmers, managers, and sponsors have to balance the forces as best they can, and still try to deliver the "right" solution. These three characteristicsimperfectly intelligent agents, exploring a vast space of possibilities, while trying to balance conflicting constraintsare the same ones that define what complexity science calls a CAS. In biology, in economics, in public policy, and in software development, the complex systems that survive and thrive behave like CAS. In human terms, organizations that survive and thrive do the same, allowing people to solve messy problems in ways that make sense, which often means muddling through a vast sea of incomplete information. Traditional software development approaches tend to ignore that. That's why a large proportion of projects fail. Managers and others in organizations need a new way to think about software development that respects the true nature of the problem. That's what growing software is. Growing software allows us to get the software we want at the end of projects, not what we thought we wanted when we started. When we start a project, we almost never know exactly what we'll end up wanting. We should stop pretending that we will. The only way that can happen is if we recognize that software development won't produce the results we want unless we think about it differently. Growing software requires a significant change in the way managers at all levels (and developers, for that matter) think about software development. It also requires different behavior. Old habits die hard, and we have half a century of conditioning to overcome. I believe it's possible, but the change can be painful. As is so often true of life, the road to better goes right through uncomfortable. But I don't believe it's hopeless if we're committed to making things better. Implications for Software Development The corporate IT world can become a radically different place if people involved with developing software can change the way they think about it. That means people who write software (programmers), people who buy software (project sponsors), and people who manage software projects (managers). This can be particularly difficult for managers, who are often under crushing pressure from above. If software development is a complex adaptive system, we cannot think of the people involved with creating it as cogs in a machine, or as workers on an assembly line. They are more than that. They are agents, albeit ones with perpetually limited information. They are interacting with other agents to create an emergent reality that is greater than the sum of the parts. Managers need to see them that way, especially since managers themselves are agents in the system as well. Those same managers need to participate in the process, rather than trying to control everything, which is much different from what they're taught and conditioned to do. What we will see is rather startling. The act of creating software to meet a specific business need involves agents with limited information exploring their way through a vast space of possibilities by following simple rules of action and interaction. That complex and unscripted dance, which often appears chaotic, can produce the kind of order that makes it possible to solve messy problems like software development. This Book Managing Software for Growth is for managers. That describes everybody in the management hierarchy, up to the CEO. But I wrote this book for people sandwiched in the middle of their organizations, the people who most often receive demands from above and somehow have to produce the demanded results. What you learned in school and on the job, the thinking you internalized, and the realities of software development make it tremendously difficult for you to make that happen. I believe it can happen, but you have to think and behave differently. I organized this book into four parts: Part I Manufacturing Software. What it means for projects, why we keep doing it, and why it makes very little sense anymore. Part II Complexity. How complexity science can help us understand software development, and be more realistic about it. Part III Growing Software. What it is, how it's different from manufacturing software, and what modern software development should look like. Part IV Change. What managers should do differently, starting tomorrow, to help their organizations grow software. My hope is that reading this book will help you Produce results that delight the people in your organization Do that without having to predict things you can't predict, or control things you can't control Participate in creating a more realistic, enjoyable place to work The point of talking about this at all is to answer one fundamental question: How can you realistically help your teams create software that people end up wanting Writing software is difficult. Writing great software is harder still. Consistently writing software that makes customers cackle with glee seems like a dream to most managers and programmers. I believe this is possible, if everyone involved is solving the right kind of problem realistically. That's what Managing Software for Growth is about.