Case Study: Should an Algorithm Tell You Who to Promote?

Aliyah Jones was having trouble paying attention to the farewell toasts. Although she was sad to see her longtime colleague Anne Bank go, she was more consumed with trying to figure out who should replace her. As a VP of sales and marketing for Becker-Birnbaum International, a global consumer products company, Aliyah knew she needed a talented marketing director to support her division’s portfolio of 34 products. After working with HR to narrow down the list of candidates, she had two finalists, both internal: Molly Ashworth, a brand manager on her team in the cleaning division, and Ed Yu, a rising star from BBI’s beauty division. Aliyah liked Molly and respected her work. Two years earlier, Molly had spearheaded a new subscription service for BBI cleaning products, which had shown strong growth in the past two quarters. Customers seemed to love the convenience, and the R&D, marketing, and executive teams had gotten excited about the service as a platform to test new offerings. Having mentored Molly through the pitch and launch of the service, Aliyah was intimately familiar with her protégé’s strengths and weaknesses and was certain that she was ready for the next challenge. But soon after the position had been posted, Christine Jenkins, a corporate VP of HR, had come to Aliyah with Ed’s résumé. Like Molly, Ed had joined BBI right out of business school and been quickly tapped as a high potential. He also had his own BBI success story: As a brand manager in the beauty group, he had revived its 20-year-old FreshFace makeup-removal product line, increasing sales by 60% in three years. Perhaps more important to Christine, he’d been recommended as a 96% match for the job1 by HR’s new people-analytics system, which she had championed. CASE STUDY CLASSROOM NOTES 1 Companies use algorithms in people-related decisions for many reasons, including consistency, reduced bias, casting a broader net, and efficiency. How might the recommendations an algorithm makes differ from those of a hiring manager who is not using data analytics? (Molly had been an 83% match.) The goal of the initiative was to expand the use of data analytics to human resources, to inform hiring, promotion, and compensation decisions.2 Aliyah was glad to see two insiders in contention—she’d come up the ranks herself—but that made the decision harder. 2 Does using algorithms for any type of people analytics violate employees’ privacy? New laws—in particular, the EU’s General Data Protection Regulation (GDPR)—are setting limits on what information employers can and cannot collect, and how employees must be notified. As the COO made a toast to Anne, Aliyah considered her interviews with Ed and Molly. Meeting Ed Yu “I’m sorry I’m so late,” Ed said, looking a little discombobulated. “My Uber driver insisted he knew a shortcut from Heathrow—but he was wrong.” Aliyah couldn’t help drawing an immediate comparison with Molly,3 who was always steady and calm, but she tried to keep an open mind. 3 Research shows that hiring managers typically form opinions about a candidate’s personality and competence in the first 30 seconds of an interview. “No problem,” she said. “Shall we get started?” “Absolutely,” Ed said eagerly. “What interests you about the job?” Ed explained that while he was proud of the growth FreshFace had seen under his leadership, he was ready for a new challenge. He’d enjoyed diving deep into one product but felt his skills were better suited for a position that would allow him to work across programs and direct a larger portfolio. Sharp, clear answer, Aliyah thought. “What have you learned in beauty that would apply in cleaning?” she asked. This was an important question. BBI’s top team had directed the divisions to share more best practices and improve collaboration. In fact, her boss wanted her to work more closely with her peers in other divisions. Ed explained how he thought his division’s approach to in-field customer research, which he credited with boosting FreshFace sales, could work in cleaning. Partnering with anthropologists was something Aliyah’s team had talked about but hadn’t yet tried out. He also asked about the new subscription program, referencing a recent white paper on trends in subscription business models. He’d clearly done his homework, was smart and ambitious, knew BBI’s business well, and seemed eager to learn. But his answers and even his questions seemed a bit stiff. Aliyah didn’t sense the dynamism or entrepreneurial mindset that she knew Molly had. Maybe he’s nervous, she thought. Or maybe that’s just who he is. Aliyah didn’t doubt Ed could do the job. But she didn’t feel excited about hiring him. Molly’s “Interview” Setting Molly’s interview up for the same day as Ed’s had seemed like a great idea when she’d suggested it to Christine, and given the noon time slot, it had been only natural to meet at their usual lunch spot near the office. But as soon as Aliyah walked into the café, she realized how unfair these back-to-backs were to Ed.4 4 Managers tend to hire people similar to themselves, studies show. For example, Kellogg School of Management professor Lauren Rivera found that managers prefer recruits who have the most potential to become friends, even over those who are more qualified. Should Aliyah worry that she’s choosing Molly because she likes her? It was impossible not to hug Molly hello and ask for a quick update on her projects and family. They even ordered the same thing: curried egg salad. But as soon as the waitress left, Molly got down to business: “I know we e-mail 10 times a day, but I’d like to treat this as a formal interview.” Aliyah smiled. “Of course.” As Christine had advised her to do, she asked questions that were the same or similar to the ones she’d asked Ed.5 5 Unstructured interviews are the default method for most hiring managers, but numerous studies have found them to be poor predictors of actual on-the-job performance. “Tell me why you’re interested in this job,” she started. It was awkward. Aliyah knew the answer already, but to Molly’s credit, she proceeded as if they weren’t close colleagues. With each response, she demonstrated deep knowledge of the business, and she had good ideas for collaborating across programs and building on the success of the subscription program. She was as polished and thoughtful as Ed, but she also seemed warmer and more self-aware. Knocked it out of the park, Aliyah thought, as they walked back to the office. Looking at the smile on Molly’s face, Aliyah knew her protégé was feeling confident that she’d done well. The Algorithm The day after Anne’s farewell party, Aliyah met with Christine and Brad Bibson, a data scientist on the people analytics team. “I know you were leaning toward Molly after we debriefed the interviews,” Christine said, “but we wanted to share some more data.” Brad handed over two colorful diagrams. “These are network analyses of Molly’s and Ed’s e-mail and meeting history at BBI. With their permission and without looking at the content of their e-mails or calendars, we analyzed who they’d been in contact with6 across the firm over the past six months.” 6 Network analyses can reveal patterns that are otherwise hard to see—for example, by identifying which employees are most central to informal information flows. It was clear from the diagrams that Ed was connected to not just his beauty division colleagues but also key people in other groups. Molly’s network was mainly within cleaning products. “I didn’t know we were doing this kind of analysis,” Aliyah said. “We’ve just started looking at networks,” Brad said, “and we think they can reveal useful insights.” “I know one chart isn’t going to sway your decision,” Christine said, “but better to have the data, right? You wouldn’t launch a new product or a new campaign without data. HR decisions should be approached the same way.” It was a pitch that Christine had made countless times while stumping for the new initiative. “We’re confident that decisions made using our algorithms are reasoned, strong, and less biased by personal feelings toward employees,” she said. Aliyah turned to Brad. “I assume you agree?” “Of course,” he said, watching for Christine’s reaction. “But as a data scientist, I also encourage healthy skepticism. Our algorithm is brand-new. We’ve used it to inform three promotion decisions so far, but it’s too early to tell how those people are doing. I don’t want to give the impression that we’re 100% confident.”7 7 Studies have revealed a phenomenon called “algorithm aversion.”Even when data-driven predictions yield higher success rate than intuitive human forecasts, people often prefer to rely on the latter. And if they learn an algorithm is imperfect, they simply won’t use it. Under what conditions would you base a decision on data analysis? Christine looked annoyed. “I appreciate your caution, Brad, but we’ve heard from the hiring managers that the type of recommendations the algorithm provides is changing the way they think about positions and candidates. And we’ve been testing the system for months now.” Aliyah sighed. “I’d trust the algorithm more if I understood it better.” She knew she wasn’t alone in her hesitation:8 Christine’s team had gotten a lot of questions about the methodology, despite the company wide training sessions. 8 Along with managers, many applicants are skeptical of algorithms, according to Pew. A majority of Americans (76%) say they would not want to apply for jobs that use a computer program to make hiring decisions. “I’d be happy to talk more about how the algorithm works,” Christine replied, “but right now you should focus on the two candidates. The point of the system isn’t to replace your judgment. The aim is to surface qualified people you wouldn’t otherwise know about so you can make a more informed decision.” “It’ll help you make a less-biased decision too,” Brad chimed in, “by relying more on the data and less on gut instinct.” Aliyah wondered whether Brad thought she was unfairly favoring Molly. She worried about that herself and cared deeply about making an objective decision. Would trusting the new system help her do that? “But the algorithm’s not completely neutral either, right?” she said. “You’re still relying on information—performance reviews, résumés—that conceivably has bias baked into it.”9 9 Data scientist Cathy O’Neil warns in her book Weapons of Math Destruction that although algorithms are fairly easy to create using historical data and can improve the efficiency of decision making, people often rely on them without understanding the biases they may be propagating. “Fair point,” Christine conceded, “and we’ve worked hard to control for that. But as a data-driven firm, we have to extend our approach to the most important part of our business: people.”10 10 How does using algorithms to analyze customers differ from using them on employees? Should companies be more cautious in implementing these methodologies internally? “It feels like you’re pushing Ed for this position,” Aliyah said. “Remember, I have to take a broader view,” Christine said. “We ran analysis to show which high potentials are at risk of leaving BBI, and Ed was near the top of the list. There is not likely to be an opening in beauty products, and we want to keep him.” “But what about Molly?” Aliyah said. “She’ll be devastated if she doesn’t get this job, and I’m sure she’d start looking too.” “Our analysis didn’t flag her as a flight risk,” Brad said. “But you could be right.” Decision Time A week later, Aliyah wasn’t any closer to a decision. She’d been avoiding Molly and had put Brad’s analyses in a drawer. Ed was clearly qualified, and he’d impressed her. But she knew intuitively that Molly was ready for the job. Did she prefer Molly because of their close relationship? Would Molly stay at BBI even if she was passed over? Aliyah needed to make a decision. Should she trust the algorithm or her instincts? Question: Should Aliyah hire Molly or Ed? The Experts Respond Prasad Setty is the head of people analytics at Google. As an enthusiast and practitioner of people analytics for many years, I believe that it is best applied as a complement to, not a substitute for, human judgment. For example, an algorithm could be used to widen the set of candidates a hiring manager might consider for a role. In the case of BBI, the people analytics system appears to have done a good job in surfacing Ed, an unexpected candidate. If the algorithm is going to make the promotion decision on Aliyah’s behalf, however, the burden of proof is very high. BBI’s track record of three decisions isn’t enough, in my opinion, to demonstrate the system’s reliability. When it comes to hiring and promotions, it is especially important to be able to explain why a particular choice is being made. Aliyah is right to want more clarity on the system’s methodology. She needs to know not just which candidate the algorithm is recommending but on what basis. So what would I suggest that Aliyah do? She should define exactly what she’s trying to achieve in filling the role—something the algorithm is not likely to know—and base her decision on that. If success for Aliyah means bringing a talented marketing director on board as soon as possible and having that person operate at the highest velocity right away, then Molly appears to be the better fit. If Aliyah is more keen on increasing collaboration with other parts of the company, then Ed, who has a wider network of relationships, seems to be the smarter choice. In either case, it’s important for Aliyah to own the decision and be able to articulate why she made a particular choice. If the algorithm is going to make the decision, the burden of proof is very high. In an ideal world, BBI would strive to make objective, unbiased hiring decisions by setting up a structured interview protocol and evaluation process, predetermining criteria, and defining what poor, mediocre, and great skills in candidates would look like. The company would advertise all roles to attract a broad applicant pool. Multiple people would conduct the evaluation, and the people making the final decision would not be the same people who evaluated the candidates. Analytics could be used to set up these protocols, recommend potential candidates, and track the impact of decisions on on-the-job performance. That’s not what’s happened in this case: Aliyah now has to select one of the candidates on the basis of BBI’s existing process, the information she has, and what her goals are. At Google, our managers don’t make hiring and promotion decisions unilaterally. All open positions are made public within the organization, and anyone is free to apply. We use independent committees to assess applicants using a structured rubric detailing what it takes to succeed in the job. We analyze the outcomes of these processes—for example, whether people thrive in their new roles—to ensure that we make high-quality people decisions. When we started Google’s people analytics group, our goal was to make all people decisions based on data and analytics. In the decade since then, we have seen some of the limits of data-driven human resources decision. Our goal now is to arm leaders with data and context so that they feel more confident in their choices, but not to undercut their role in the process. Today, our team’s mantra is to help all people make decisions based on data and analytics. Patty McCord is the former chief talent officer at Netflix and an adviser to start-ups and entrepreneurs. Aliyah should trust her instincts, not the algorithm. Only she knows what she truly needs from a director of marketing, and it’s clear that Molly has it. Before I left Netflix, I was pitched by a lot of data analytics companies that promised to help us make better people decisions using their algorithms, but I didn’t see the ROI. At the time, we employed a few thousand people, and the cream was already rising to the top, so it was hard to imagine how an algorithm would significantly improve our decisions. BBI is a much bigger company, so it’s possible that AI could be more useful in that context. But for now, the decision should rest with Aliyah. Of course, she needs to follow best practices for hiring managers and look at the team as a whole in her analysis. What is its current makeup, and how will the newly promoted director fit in? What results would Aliyah expect to see in six months to a year to demonstrate that the division is working well? And with the proper support, which candidate is most likely to achieve those results? Aliyah is also right to question her own bias toward a woman she considers a protégé and with whom she has worked closely for several years. Hiring managers often say, “I’m looking for someone who is smart, solid on their feet, and ready to jump into the role,” and it just so happens that the person they like best looks just like them. This isn’t always overt bias. It’s often a matter of not wanting to take a risk. Christine says the algorithm is unbiased, but we can’t be sure she’s right. Although I don’t see any evidence that Aliyah is favoring Molly because she’s a woman, I do think gender should be a consideration in this case. Research has repeatedly demonstrated that women are not promoted at the same rate as men. Throughout my HR career, I’ve seen men consider female candidates but hire a man instead, telling me, “I know she’s qualified, but I’m looking for someone who is ready to step up now, and I don’t want to set her up to fail.” Of course, you can’t be set up to win if no one gives you the opportunity to fail. I worry that if Molly doesn’t get this chance, she might not get another. Christine touts her algorithm as unbiased, but without more information about the methodology and the criteria it uses, we can’t be sure she’s right. In fact, I’m particularly suspicious of the network data Brad brought to the meeting. Could it be that Ed is more connected—that he’s invited to more meetings, in touch with more people—because he’s a man and has been given more opportunities to shine? If I were Aliyah, I’d ask what those network maps look like for men and women across the company. I’ll admit that if the gender roles were reversed in this case, with a male hiring manager favoring a male candidate he knew well over a woman recommended by an algorithm, my advice would probably be different. In this case, the algorithm has done its job in recommending an unexpected man. But if Aliyah has conducted a fair analysis—consciously setting her preconceptions aside and objectively considering who is best for the job—and she still prefers Molly, she should trust her gut and promote her. HBR’s fictionalized case studies present problems faced by leaders in real companies and offer solutions from experts. This one is based on the HBS Case Study “Susan Cassidy at Bertram Gilman International” (case no. 417053), by Jeffrey T. Polzer and Michael Norris.

Case Study: Should an Algorithm Tell You Who to Promote?

Summary.

Partner Center