HBR Case Study: A Rush to Failure?

There is absolutely no reason why the contractors shouldn’t be able to give us rapid product development and flawless products—speed and quality both,” David MacDonagle said as he tried to light a cigarette. The warm wind, portending rain, kept blowing out his matches. Finally he gave up and slipped the cigarette back in his pocket. MacDonagle, the head of the Canadian Aeronautics Administration, was nervous. Everyone at CAA headquarters was nervous. Very shortly, the project that many of them had devoted the past four years to would have its first real-world test, 350 kilometers above the earth. Feeling cooped up in the executive offices and oppressed by the presence of the media, MacDonagle had gone outside to breathe some air—actually, some tobacco smoke—and had invited the sharp young program manager Samantha Van Sant to join him. Van Sant, a former Canadian army major, had a lot of skin in the project too. Since 2006 she’d been managing the two contractors the CAA had commissioned to build the $1.2 billion set of giant robotic arms known as Retractable Extended-Arms Compatible Holder, or REACH, for the International Space Station. “So how do you deal with nerves?” MacDonagle asked. “I usually go out for a run,” Van Sant said, looking down the road that led from CAA headquarters through the cornfields, on which she’d logged many miles. They turned to look back at the agency’s buildings, which despite their grandeur looked small in the empty Quebec landscape. The sight reminded Van Sant of one of MacDonagle’s catchphrases: “We are a small spacefaring nation….” Canada was indeed a small player in space compared with the U.S., Russia, Europe, and Japan. Always at risk of being marginalized, the CAA had done everything possible to get the REACH contractors, Hollenbeck Aircraft and Eskina Software Systems, to complete the first phase of the project in time to get it to the space station this year, when the orbiting lab would officially be complete. And, amazingly, they had made the deadline—and come in on budget. REACH was now attached to the station, though there was still much more to come, including an even more sophisticated set of “hands” that would fit on the ends of the robotic arms for extremely delicate work. The additions were to continue for two more years. The contractors had been great about speed; the problem was quality. Glitches with the software, motors, and circuits had kept turning up. The fact was, not a single test in four years had gone flawlessly. “Yeah, yeah, we can fix that,” the contractors’ reps always said, dismissing the CAA’s concerns. “Hey, this is life in the fast lane,” a rep told Van Sant after one of REACH’s arms had failed to retract on command. “Remember, we told you that the compressed schedule would increase the risk.” The contract that she managed called for parallel development, meaning that the project’s phases—R&D, prototyping, testing, production, and quality control—overlapped, with each one beginning while the previous one was as much as 50% incomplete. That was sacrilege in some aerospace circles. But owing to the space station’s construction deadline and the ever-present threat of cuts to the CAA’s budget, the agency was aiming to do a decade’s worth of work in six years. Computer simulations had to take the place of some real-world testing. Component quality control was less thorough. Because of all the unknowns in the project, the CAA had agreed to a cost-plus-fixed-fee contract, under which the contractors were paid a specified amount over their costs for labor, materials, and overhead. MacDonagle’s insistence on a rapid approach to development had been one of the main reasons Van Sant had been hired as a program manager. During her years in the army, she had established a bulletproof reputation for being aggressive and goal oriented. She and MacDonagle saw eye-to-eye. She knew speed was critical. “I know what the media hounds will ask me: Is REACH going to work this time?” MacDonagle said. “We’d better go, I guess,” MacDonagle said. “The media hounds are waiting. I told them I’d do quickie interviews once I got back. I know what they’ll ask me: Is REACH going to work this time?” As they headed toward the building, the rain started. He looked at Van Sant. “So is it?” Trouble in the Air Red marker in hand, MacDonagle held forth before a group of reporters, asking whether they were aware that 50 years ago, the U.S. had blasted half a billion inch-long copper needles into orbit to reflect radio waves and thereby facilitate communications. Those needles were still floating around, and some had torn through one of the space station’s solar collectors. “The solar arrays are the big bird’s big red wings,” he said, turning to the whiteboard and drawing the collectors. He drew a gash in one of them. “A hole here means less electricity,” he said, tapping the board. “Ever since the solar array got that hole from those flying needles, the space station has been operating on less power. Fixing it is tricky, because it’s very far from the modules where the astronauts work and because of the risk of electrocution. Once an array is in place, you can’t turn it off. It keeps generating power from sunlight. So if a spacewalker were to try to go out there and fix it, he’d be liable to get 100 volts of direct current through his body. That’s where REACH comes in.” He took a moment to draw the Canadian creation, then stepped back to admire his sketch. The machine’s two long arms stretched out toward the solar array in a nurturing embrace. “Fixing solar arrays isn’t what REACH was designed for,” MacDonagle said. “It’s meant to do the mundane work of replacing battery units on the exterior of the space station. But since REACH is up there, it’s being pressed into service for the repair. It will stitch together the solar array while the astronauts control it from the safety of their module.” He put the cap back on the marker and began fielding questions. As Van Sant watched, someone tapped her on the arm. It was Alfred Siroy, the head of a CAA panel that had been trying to find out why there were so many quality issues with equipment from the Hollenbeck-Eskina venture. “How long before they deploy it?” he asked. “Soon—later—I’m not sure,” she said. Siroy always made Van Sant defensive. He’d made no secret of his disapproval of the way the REACH program was being managed, and of the parallel approach in particular. She knew that this viewpoint would figure in his forthcoming report. Fortunately, he was an overly meticulous writer, so the draft was taking forever. She asked him how the writing was going. “Slow,” he said, shaking his head. “But we do have a title: ‘The Rush to Failure.’” This gave her a start. “What failure?” she asked. “REACH is about to perform a critical repair task.” He shot her a skeptical look. “Rapid ramp-up was a laudable goal,” he said. “But you have to give a contractor adequate time for QA. And you have to have a contract that gets the incentives right.” He continued: “We did an analysis. Your compressed schedule forces Hollenbeck-Eskina to cut corners, resulting in prototypes that fail. Substituting computer simulations for rigorous ground testing is a recipe for disaster. There isn’t an electronic data management system that would allow the contractors and the CAA to access current test data for analysis. The prototypes aren’t equipped with the instruments that would provide adequate test data. And the contractors have no incentive to push back: The cost-plus contract puts all the risk on the CAA. The agency and the contractors have different goals and objectives.” Van Sant couldn’t disagree about the contract; its weaknesses had become increasingly evident to her. But if speed was the priority, they were unavoidable. “The contract language is ancient history,” she said dismissively. “You can rewrite history,” Siroy said. “Any contract can be altered as long as both sides agree—you know that.” Suddenly, Van Sant saw, one of the staff members who’d been monitoring the goings-on at the space station began ushering MacDonagle away from the reporters, who, smelling blood, tried to follow. MacDonagle caught her eye, and Van Sant didn’t like the look she saw on his face. Something bad had happened. She slipped inside the communications room, where journalists weren’t allowed, just before the door was shut. Over the speakers she could hear the astronauts at the space station talking about the power switching unit and using the word “failure.” She heard someone say, “We have to go to Plan B.” REACH was probably experiencing the same problem that had come up during its last on-ground test. A system failure notification had gone off, but the contractors had dismissed it as a “reporting error,” meaning it hadn’t reflected a true mechanical breakdown. Still, no one wanted to deploy REACH while red lights were flashing. There was too great a risk that the robotic arms would fail at a critical moment. “What’s Plan B?” Van Sant asked MacDonagle. “I don’t know,” he said quietly, “but whatever it is, it won’t involve anything that came from us. REACH is Canada’s only contribution to the space station.” Support from on High “We’ve got video!” someone shouted, and there on the screens were multiple images of a man in a space suit dangling at the end of a loading crane. It was well past midnight, but the reporters were still at CAA headquarters. They gathered around the screens. MacDonagle and Van Sant had long since given up trying to avoid them and were mingling with them as the repair attempt unfolded high above West Africa. Everyone watched in silence as the crane, jury-rigged for the purpose, carried a spacewalker toward the space station’s torn solar array. REACH couldn’t even be seen—it was docked somewhere else. “Harris Webb,” MacDonagle said ruefully as he watched the figure in the shiny suit. “It’s so fitting.” Years ago, MacDonagle and Webb had been pilots on the same shuttle mission, and Webb, though much younger, had been chosen to lead it. A U.S. physician, mountaineer, author, pilot, and gourmet cook, as well as an astronaut, he was the ultimate go-to guy, brilliant and fearless—almost to the point of being foolhardy. The reporters were excitedly discussing his stunt. Because REACH had failed, Webb was going out on the end of the crane to repair the array by hand. In one gloved hand he held what looked like an oversized hockey stick wrapped in insulation, so that he could stop himself from bumping into the arrays. In the other he carried several two-meter lengths of plastic cable that would be used to “stitch” the pliable solar array back together. “Wow,” a reporter gasped as Webb, reaching awkwardly, began threading one of the cables through the openings in the array. “Cowboy,” MacDonagle hissed under his breath. He put his hands on his head and looked at the ceiling. Van Sant spotted Charlie Truss, one of the reps from Hollenbeck, sitting in a corner, his tie loose. He looked miserable. But she didn’t feel pity for him—just annoyance. She went over to him. “Whatever happens up there tonight,” she said, “things are going to change down here. Our only way forward from this fiasco is to show that we’ve taken concrete steps to improve QA and finally get some positive results.” “I’m all for that,” Truss said. “But anything you do to increase QA is going to slow things down. Once that happens, the costs start increasing and you become vulnerable to budget cuts. If we turn our existing contract into a traditional aerospace contract with all those sequential steps and inevitable delays, we might as well say good-bye to the improvements to REACH that are in the pipeline.” “If speed has to be sacrificed for a more reliable REACH, then so be it,” Van Sant said. “The contract has major flaws. You’re accountable for speed but not performance. We have to share the risk. We need a contract with performance-based incentives and penalties so that we can balance speed, quality, and results. Our goal is reliable components and systems that perform—and that should be your goal, too. We need to work as a unified team willing to push back on each other to get results.” Behind her the reporters gasped. She rushed to a screen and was relieved to see only that Webb, as he had finished his repair, had accidentally let go of a set of pliers, which was now drifting off into space. But then Webb did something incredible. He turned to his Earth audience and began making a statement defending the failed REACH. “Everyone’s going to blame REACH, but they shouldn’t,” he said. “It’s a great piece of technology. I want to commend David MacDonagle and the CAA for overcoming a lot of technical obstacles in a big hurry and getting REACH up here on time. It’s going to be a vital part of our operations. One little power-unit problem doesn’t mean anything. Complex machines fail—that’s just the way it is. We’ll fix it, just like we fixed the solar array. I understand that the CAA has a great upgrade coming in the next couple of months—a new set of robot hands that are so nimble they can peel a hard-boiled egg. I say, Get it ready and shoot it up here. We’ll start using it right away. I’ve got a few eggs that need peeling.” Van Sant was stunned, and she could see that Truss was, too. Then, giving her a small smile, he asked: “What were you saying about the contract?” What Would You Do? Some advice from the HBR.org community Yes, renogotiate. This contract is rife with problems. The project’s phases overlap, and real-world testing is minimal. Most important, the project failed. Webb saved the agency’s reputation and tempered the storm in the short term with his surprise comments. Now the CAA has to rework the contract to produce a successful product, because without one it will lose all momentum in its space efforts. Tom Geiser,managing partner, Black Twig Communications Alfred Siroy offers excellent outside expertise. Samantha Van Sant and Charlie Truss would do well to seek out his advice, as he’s interested in seeing REACH thrive but through a different approach that would not sacrifice quality. Contract reviewing is essential—but getting the right people in the project is the key success factor for REACH. Bogdan M. Negru,visiting lecturer, Griffiths School of Management, Emanuel University of Oradea It’s time to renegotiate, but the contractor would never accept a pure flat-fee arrangement, given the potential exposure. A cost-plus structure, with kickers for quality and speed and penalties for delays and problems, would probably work best. The project’s very ambitious scope might be worth revisiting, too. Focus first on getting the basic arms working and later perhaps on making the hands capable of peeling a banana, not an egg. Tom Leung,senior vice president, Marchex Should Van Sant push for a renegotiated contract for REACH? Gary L. Moe is a director in McKinsey’s Business Technology Office and is based in Silicon Valley. Whenever you hear about large, complex, costly government-sponsored tech initiatives that fail to meet expectations, the blame almost always falls on the “hurry up” schedule. If only the public agency hadn’t pushed the contractors so hard, if only the developers had been given more time to refine the design, if only a few more months or years had been built into the production schedule, the technology would have worked perfectly. But it’s not true. Government sponsors can give contractors all the time and money in the world to complete a detailed requirements analysis and perfect a project design, and the finished product still probably won’t work flawlessly. So-called big-bang design, in which developers labor mightily to get every last detail correct, is incapable of yielding a fully functioning, error-free product, because human beings, no matter how brilliant they are, cannot foresee all the issues that might arise in a complex technology. Human beings, no matter how brilliant, cannot foresee all the issues that might arise in a complex project. What’s needed instead is an iterative development process, whereby you build a prototype or even a fully fledged product and then put it out there, test it, and learn from its weaknesses, most of which you couldn’t have seen on the drawing board or in a 3-D simulation. In the next iteration, you enhance the product. Then you run another round and another and another until finally you have something that works. The more complex a technology is, the more iterative the development needs to be. This is how products are developed in the nongovernment sector. Take the auto industry. Cars are so complex that a manufacturer will put a model through a number of builds and a lot of testing before starting full production. In software, this process is called agile development. Developers write code for a week, test it to find out how to improve it, and then write some more code. An interactive approach also works best when it comes to organizational change. No matter how hard you work on designing a reorganization, you will get it only 60% right, so you have to keep working at it, and finally you’ll get to 90% right. (It never gets any better than that.) Iterative development can work both for big long-term projects and for big one-offs, which is what many components of the International Space Station are. You break the development into pieces and apply the build-test, build-test method to the parts. So why do government organizations, especially those that are defense- or space-related, favor the big-bang approach? There are a number of reasons, some involving the way procurement is handled. But the root of the problem is that iterative development entails experimentation, and experimentation entails failure. Government agencies don’t like failure because it ruins political careers. So they try to avoid experimentation. But usually what they end up with is an even bigger failure—and no clue about why it happened. My advice for Samantha Van Sant, then, is to restructure the contract to break further development of REACH into small chunks and require the contractors to practice iterative improvement, instead of striving for full initial functionality. Cutting down on the functionality delivered in each phase might help too, since 80% of cost and schedule overruns are usually due to the last 10% to 20% of requested functionality. Tom Quinly is president of Curtiss-Wright Controls, based in Charlotte, North Carolina, which develops products for aerospace, defense, and industrial markets. In the world of cutting-edge product development, the struggle between speed and quality is over. Speed has won—decisively. In today’s highly competitive global markets, getting innovations out quickly can mean the difference between success and failure. But it’s also a given that the quality must be high. Quality has become table stakes. The struggle between speed and quality is over. Speed has won—decisively. A lot of research has been published on the holy grail of lightning-speed development. Concurrent engineering programs, agile teams, risk mitigation programs, spiral development, outsourcing, harmonizing tools, and advanced simulation and modeling tools all can help you attain it, but your own people, processes, and market demands will determine the right recipe. What works best for us is forming small, seasoned, highly talented teams; being clear about time-to-market expectations; making sure developers have the right set of tools; and keeping our technical teams engaged, customer focused, and happy. Bureaucracy, however, is an innovation killer. It’s inevitable that as a business grows, things that don’t add value creep into processes. With each slipup there’s a tendency to add another process check. In isolation, each makes perfect sense, but in the aggregate, innovation is choked, and the team can’t move nimbly. I recall an experience early in my career, when a major development project had gone poorly. At an executive review, I was prepared to explain what had happened, what we’d learned, and what we’d done to stop the bleeding. Our CEO looked around at the others in attendance and said that he was disappointed—not in the group that had failed (mine), but in the other groups, because they hadn’t failed. They weren’t being as aggressive as he expected. That story lives on in the lore of the company, and it says a lot about our culture. It’s not that we encourage failure—we use it as a tool. Complex development programs rarely (if ever) come out of the gate perfectly; something unanticipated often happens. A lot can be learned from reviewing failures and looking at processes, expectations, people, and tools. Were our requirements too ambitious or ill defined? Did we not map the highest-risk areas and have well-designed mitigation plans? This relentless focus on learning improves predictability, reduces cycle time, and helps us get a high-quality offering to market ahead of the competition. That’s why I would advise Van Sant to give up the dinosaur perspective that speed means having to sacrifice quality. She should engage the entire team in examining the failures and exploring ways to achieve quality without upsetting the schedule. If REACH has few qualified alternative partners, then disrupting a long-standing contract with a highly experienced and specialized partner would be counterproductive. If Hollenbeck-Eskina is the best option, I would avoid having to reopen the contract and potentially lose the partner’s deep project knowledge. However, if Hollenbeck-Eskina was not totally engaged and forthcoming in helping understand and correct the failures, it may be time to involve other potential partners or to stand firm on a contract renegotiation.

HBR Case Study: A Rush to Failure?

Partner Center