Humans Are Still Better Than AI at StarCraft—for Now

In the computer game StarCraft, humans still have an edge over artificial intelligence.

That was clear on Tuesday after professional StarCraft player Song Byung-gu defeated four different bots in the first contest to pit AI systems against pros in live bouts of the game. One of the bots, dubbed “CherryPi,” was developed by Facebook’s AI research lab. The other bots came from Australia, Norway, and Korea.

The contest took place at Sejong University in Seoul, Korea, which has hosted annual StarCraft AI competitions since 2010. Those previous events matched AI systems against each other (rather than against humans) and were organized, in part, by the Institute of Electrical and Electronics Engineers (IEEE), a U.S.-based engineering association.

Though it has not attracted as much global scrutiny as the March 2016 tournament between Alphabet’s AlphaGo bot and a human Go champion, the recent Sejong competition is significant because the AI research community considers StarCraft a particularly difficult game for bots to master. Following AlphaGo’s lopsided victory over Lee Sedol last year, and other AI achievements in chess and Atari video games, attention shifted to whether bots could also defeat humans in real-time games such as StarCraft.

Unlike Go, which allows bots and human players to see the main board and devote time to formulating a strategy, StarCraft requires players to use their memory, devise their strategy, and plan ahead simultaneously, all inside a constrained, simulated world. As a result, researchers view StarCraft as an efficient tool to help AI advance.

A number of professional StarCraft gamers have said they welcome the challenge of playing against bots. Two leading players told MIT Technology Review earlier this year that they were willing to fight bots on broadcast TV, as in the AlphaGo match, if asked. Executives at Alphabet’s AI-focused division, DeepMind, have hinted that they are interested in organizing such a competition in the future.

The event wouldn’t be much of a contest if it were held now. During the Sejong competition, Song, who ranks among the best StarCraft players globally, trounced all four bots involved in less than 27 minutes total. (The longest match lasted about 10 and a half minutes; the shortest, just four and a half.) That was true even though the bots were able to move much faster and control multiple tasks at the same time. At one point, the StarCraft bot developed in Norway was completing 19,000 actions per minute. Most professional StarCraft players can’t make more than a few hundred moves a minute.

Song, 29, said the bots approached the game differently from the way humans do. “We professional gamers initiate combat only when we stand a chance of victory with our army and unit-control skills,” he said in a post-competition interview with MIT Technology Review. In contrast, the bots tried to keep their units alive without making any bold decisions. (In StarCraft, players have to destroy all of their competitors’ resources by scouting and patrolling opponents’ territory and implementing battle strategies.)

Song did find the bots impressive on some level. “The way they managed their units when they defended against my attacks was stunning at some points,” he said.

Kim Kyung-joong, the Sejong University computer engineering professor who organized the competition, said the bots were constrained, in part, by the lack of widely available training data related to StarCraft. “AlphaGo improved its competitiveness and saw progress by learning from data [about the game Go],” Kim pointed out.

That will change soon. In August, DeepMind and the games company Blizzard Entertainment released a long-awaited set of AI development tools compatible with StarCraft II, the version of the game that is most popular among professional players.

Other experts now predict that bots will be able to vanquish professional StarCraft players once they are trained properly. “When AI bots are equipped with [high-level] decision-making systems like AlphaGo, humans will never be able to win,” says Jung Han-min, a computer science and engineering professor at the University of Science and Technology in Korea.