SerpentAI

It works with any game that runs on a desktop and treats the game as a black-box environment observable only through screen pixels and controllable only through simulated input.

The framework includes game environment plugins for communicating with the game process, a visual game state analyzer for detecting sprites and game elements via template matching and feature detection, an input controller for simulating keyboard and mouse actions, and integration hooks for reinforcement learning algorithms.

Serpent.AI handles the operational complexity of game-based RL research maintaining frame capture rates, synchronizing agent actions with game state updates, logging training progress so researchers can focus on algorithm design rather than game integration plumbing.

Serpent.AI is open-source under the MIT license and supports Windows, macOS, and Linux, running with any game regardless of engine or platform.

It is used by AI researchers studying reinforcement learning in complex visual environments, game AI enthusiasts building bots for personal projects, and educators demonstrating RL concepts through the tangible, visually intuitive medium of game play.

The framework's game-agnostic design means skills developed working with one game transfer directly to creating agents for other games.