Contention around the implications of artificial intelligence continues to grow. While workers worry automated agents will replace them, and business leaders ponder what tasks AI can fulfill best, one less-discussed topic is how well automated agents and human professionals work alongside each other in teams.
Bruce Kogut, the Sanford C. Bernstein & Co. Professor of Leadership and Ethics at Columbia Business School, along with Fabrizio Dell’Acqua, PhD ’22, and Patryk Perkowski, PhD ’23, decided to dig into this question. The trio examined how substituting workers with AI avatars on teams impacts performance and coordination, as well as the behaviors and efforts of the remaining human coworkers.
“AI is here to stay. Firms are rapidly trying to figure out how to use it and integrate it into the workforce. It is already part of teams in many places, so we should start preparing our students for it,” says Kogut, who has taught at Columbia for 15 years. “It’s not just a question of whether you’re going to replace workers with AI but also whether you can have AI and people work together in a team and be effective.”
How well workers can team up with chatbots, robots, and other AI labor has broad implications for businesses and governments around the world as they grapple with how to introduce this technology into their workforces.
Adding AI to Teams Can Hurt Performance
In their research, “Super Mario Meets AI: Experimental Effects of Automation and Skills on Team Performance and Coordination,” set to be published in the Review of Economics and Statistics, the authors use video games as their testing ground. First, participants' skills were tested on a set of video games to assess their playing abilities. Then, they were asked to play a Super Mario game called “Dine and Dash” on the Nintendo Switch console. Teams had one minute to collect certain ingredients to make a recipe. The task has the property of many real-world jobs, which often require team communication, coordination, and strategic planning. The game has a built-in AI player that can outperform the vast majority of human players, allowing the researchers to study the impact of AI on team and worker performance.
The study found that individual human workers became less productive after AI became part of their team. It also found that overall team performance suffered when an automated worker was added. With an AI player onboard, humans proved less adept at anticipating one another’s movements, resulting in them bumping into other team members more and coordinating less than with human-only teams.
Performance declined significantly in the low- to medium-skilled teams. Highly skilled teams, on the other hand, were not negatively impacted by AI's introduction. It seems that the routines used by highly skilled teams allowed them to more easily absorb automated workers.
Humans Dislike Working with AI Agents
Regardless of skill level, workers don’t enjoy working with AI, even when the automated agents are more productive than the human workers and earn them a higher incentive bonus. In surveys, the study participants said they lacked trust in AI and reported high levels of suspicion, even more so after playing with these agents.
“Humans have an aversion towards AI,” says Kogut, adding that other studies have found that as well. For example, recent research from CBS Professors Miklos Sarvary and Bernd Schmitt found that American workers strongly dislike robots when they closely replicate human interactions. Japanese workers, however, were less bothered.
Kogut says it will be interesting to study worker response as machines learn to exhibit emotions, such as how to socialize, laugh, or empathize, something they are being trained to do now.
Watch our recent interview with Professor Hod Lipson, an AI expert and director of the Creative Machines Lab at Columbia University:
Implications for Business
Leaders hoping to incorporate AI labor will need to think about more than simply building the technology.
“The results indicate that introduction of intelligent machines in firms is likely to have relevant and potentially detrimental consequences on culture and employee motivation through the effects on human workers,” the authors write. As AI becomes more and more capable, human workers may defer to the technology and exert less effort or even grow less confident in their own skills.
Leaders also should understand that the strategies needed to successfully integrate AI may differ across groups. “High-skilled workers are more successful in integrating their automated coworkers,” says Kogut. “So AI won’t affect all workers the same. There will be some who do better.” Firms may have an easier time integrating AI labor if they also incorporate a set of written protocols and practices, he adds.
Kogut notes this differentiated approach may also apply to global organizations, which may need to tailor their AI strategies for the various regions in which they operate. For instance, leaders may need to create different protocols, training, and messaging for Japanese and American workers, as Sarvary and Schmitt’s research shows they react differently to AI.
Up Next for Kogut
No surprise, interest in AI is high and keeping Kogut busy. In May, he helped lead the 2nd Annual Management, Analytics, and Data Conference, which included panel discussions on the impact of AI on work and organizations. And at CBS’s new think tank, The Hub, he oversees the Business, AI, and Democracy (BAID) initiative and is already planning the group’s next conference.
Kogut is also working on AI experiments involving self-driving cars, attempting to identify what conditions humans are better at handling, the circumstances when AI is more equipped, and how to optimize both of their skill sets together.
“Right now is a great time to be running experiments about human and AI interaction. There is so much to learn,” Kogut says. “And it’s a great time to be working in teams when my co-authors are human — or so they claim.”