Imagine having a gaming companion who’s good at playing various 3D video games and responds to your verbal instructions with the eagerness of a human teammate. Google DeepMind has taken a significant leap towards this future with their latest innovation, SIMA (Scalable Instructable Multiworld Agent), an AI designed to play multiple video games and act upon your commands, pushing the boundaries of traditional game-playing AI models.
A leap beyond conventional game AI
Unlike traditional AI characters in games, which operate within predefined rules and commands, SIMA has no direct access to the game’s code or mechanics. Instead, it learns from observing hours of human gameplay across different games. This observation includes the actions taken and verbal instructions shared between players, providing a rich dataset from which the AI can understand and replicate human-like gameplay.
For example, SIMA can learn that moving pixels in a specific pattern equates to “moving forward” or that interacting with an object resembling a doorknob results in a door opening. This approach allows it to understand and perform actions more nuanced than the binary outcomes of winning or losing, typical of most game AI.
Bridging games and the real world
The training for SIMA wasn’t limited to one game. Instead, it encompassed a variety of titles, from “Valheim” to “Goat Simulator 3”, with the cooperation of their developers. This diversity in training aimed to equip the AI with the ability to generalise its learning to games it has never encountered before, a concept known as generalisation in AI research. While the AI performed better on unfamiliar games than those trained on single titles, each game’s unique mechanics and terminologies still pose a challenge, highlighting the need for extensive and varied training data.
One of the project’s leads, Tim Harley, emphasised the goal of creating a game-playing companion that feels more natural and less rigid than the AI characters we’re accustomed to. By observing and learning from human gameplay, SIMA is designed to adapt and respond in ways more aligned with how humans play and interact with games.
Beyond gaming: The future of AI companions
This approach to training AI opens up new possibilities beyond gaming. Traditional AI training methods like reinforcement learning rely on clear reward signals like scores or win/loss outcomes. However, using imitation learning from human behaviour, SIMA is trained to pursue objectives based on textual goals. This allows for a broader range of actions and interactions that more closely mimic human decision-making.
The potential for AI agents like SIMA extends into other domains where open-ended collaboration and creativity are valued. From enhancing NPC interactions in games to simulating complex behaviours, the technology behind SIMA represents a step towards more versatile and human-like AI companions.
As we look towards a future where AI can serve as a cooperative companion in various tasks, the work by Google DeepMind’s researchers on SIMA offers a glimpse into the exciting possibilities of blending AI technology with human-like intuition and adaptability.
