A Turing Test for Computer Game Bots

Can a computer fool expert gamers into believing it’s one of them? That was the question posed at the second annual BotPrize, a three-month contest that concluded today at the IEEE Computational Symposium on Intelligence and Games in Milan.

The contest challenges programmers to create a software “bot” to control a game character that can pass for human, as judged by a panel of experts. The goal is not only to improve AI in entertainment, but also to fuel advances in non-gaming applications of AI. The BotPrize challenge is a variant of the Turing test, devised by Alan Turing, which challenges a machine to convince a panel of judges that it is a human in a text-only conversation.

“The BotPrize is important for AI in gaming because it aims to show how AI can make games more fun to play, by providing more interesting opponents for game players,” says Philip Hingston, associate professor in the School of Computer and Information Science at Edith Cowan University in Perth, Australia, and an overseer of the competition. “It is also important for AI in general because it highlights a central question in AI: How is human intelligence related to computer intelligence?”

This year’s BotPrize drew 15 entrants from Japan, the United Kingdom, the United States, Italy, Spain, Brazil, and Canada. Entrants created bots for Unreal Tournament 2004, a first-person shoot-‘em-up in which gamers compete against each other for the most virtual kills. For the contest, in-game chatting was disabled so that bots could be evaluated for their so-called “humanness” by “physical” behavior alone. And, to elicit more spontaneity, contestants were given weapons that behaved differently from the ones ordinarily used in the game.

Each expert judge on the prize panel took turns shooting against two unidentified opponents-one human-controlled, the other a bot created by a contestant. After 10 to 15 minutes, the judge tried to identify the AI. To win the big prize, worth $6,000, a bot had to fool at least 80% of the judges. As in last year’s competition, however, none of the participants was able to pull off this feat. A minor award worth $1,700, for the most “human-like” bot, was awarded to Jeremy Cathran, from the University of Southern California, for his entry, called sqlitebot.

Artificial intelligence has long been crucial to creating convincing and compelling computer games, whether a player is competing against drivers in Mario Kart on the Nintendo Wii or alien invaders in Halo 3 for the Microsoft’s Xbox 360 games console. And, as competition increases in the $21 billion game industry, developers are striving to make game AI even more convincing. But creating a good bot presents a formidable challenge, says Steve Polge, lead programmer of Epic Games, the company that created Unreal Tournament. “You don’t always want your AI to perform just like a human,” he says. “Humans can be pretty annoying and obnoxious opponents.” Instead, Polge says, developers often strive for “AI that can make unexpected plans and present emergent and surprising challenges to the player, which will definitely lead to better games.”

Risto Miikkulainen, a professor of computer science and neuroscience at the University of Texas at Austin, was among the BotPrize participants who tried to concoct just the right mix of human and machine. When coding a bot for this year’s contest, Miikkulainen and his team designed the bot to learn quickly. “When humans play games, they adapt very quickly,” he says, “so in creating a bot, you can’t aim to be 100% accurate, because adaptation is inexact.”

The BotPrize is an attempt not only to improve game technology, but also to foster innovations outside the industry, from AI used in emergency training simulations today to the companion robots of the future. “You need some way to measure milestones in AI research,” says Robert Epstein, creator and former director of the annual Loebner Prize Competition in Artificial Intelligence, which involves a conventional Turing test. “So when you arrange contests like the BotPrize, you have a way of knowing whether we reached a milestone.”

Will Wright, creator of best-selling simulation games such as The Sims and Spore, hopes the BotPrize encourages AI researchers to pursue the most elusively human quality of all: emotion. “Machine interactions are becoming a ubiquitous part of our environment, but they’re not necessarily the most satisfying,” Wright says, “so acknowledging our emotional dimension is an interesting task to go for in AI.”

This means developing bots that not only fool people but also move them emotionally. “You want to build an emotional model for the agent you’re competing with,” Wright says. “It’s not just about having an accurate aim. It’s about creating a bot that simulates a victory dance above your dead corpse.”