An AI Ally to Combat Bullying in Virtual Worlds

In any fictionalized universe, the distinction between playful antagonism and earnest harassment can be difficult to discern. Name-calling between friends playing a video game together is often a form of camaraderie. Between strangers, however, similar words assume a different, more troublesome quality. Being able to distinguish between the two is crucial for any video-game maker that wants to foster a welcoming community.

Spirit AI hopes to help developers support players and discourage bullying behavior with an abuse detection and intervention system called Ally. The software monitors interactions between players—what people are saying to each other and how they are behaving—through the available actions within a game or social platform. It’s able to detect verbal harassment and also nonverbal provocation—for example, one player stalking another’s avatar or abusing reporting tools.

“We’re looking at interaction patterns, combined with natural-language classifiers, rather than relying on a list of individual keywords,” explains Ruxandra Dariescu, one of Ally’s developers. “Harassment is a nuanced problem.”

When Ally identifies potentially abusive behavior, it checks to see if the potential abuser and the other player have had previous interactions. Where Ally differs from existing moderation software is that rather than simply send an alert to the game’s developers, it is able to send a computer-controlled virtual character to check in with the player—one that, through Spirit AI’s natural-language tools, is able to converse in the game’s tone and style (see “A Video-Game Algorithm to Solve Online Abuse”).

Ally’s bot might say: “I noticed you and Player 42 seem to be strangers. They may have said something inappropriate to you, so I just wanted to check if you’re okay?” The player is then able to respond to the bot, Dariescu says. If the interaction is dismissed as friendly banter, the system registers this and updates its rules so it won’t intervene over similar behavior in the future. If the user feels abused, the system intervenes with the culprit.

Ally is compatible with all online games, and Spirit AI is working on adding capabilities to police behavior in virtual-reality games (see “The War on the Disturbingly Real Trolls in Virtual Reality”). The tool can be configured in a variety of ways to interact with players and take different kinds of action. Spirit AI sees Ally as a way to augment a game’s community management team, which can monitor activity in a dashboard.

The company, which was officially formed in 2016 and currently has a staff of 15, won’t disclose which game developers it is currently working with, but the software has attracted attention in the gaming world. “On the simplest level, AI-supported community management tools can extend the team’s reach,” says Imre Jele, a veteran designer of vast online multiplayer games such as RuneScape. “But what I’m most excited about is AI systems that go beyond identifying and punishing rule-breaking behavior. Ultimately the goal of these AI team members should be to get out of policing and into shaping positive interactions and building healthier communities.”

Spirit certainly intends for the software to offer more than a mere digitized police force that can be summoned into online games. “We’ve identified the potential for this technology to not only prevent bad behavior but also encourage good, prosocial behavior,” says Dariescu. The software could also use ambient eavesdropping to identify broader issues that players might have with a game’s design, providing insights to the developer as to how the overall player experience might be improved.

There are, however, limitations when it comes to AI interventions in cases of harassment, according to Jele. Many interactions between community management and player support teams require deep empathy and sympathy, he says: “The players supported the developers with their money and deserve the best possible care, which only humans can provide at the moment.”