Teaching a Spaceship to Play My Game: Running a PPO Agent Inside the Real, Unmodified Game

The Game the Agent Has to Learn

In AlifallX you fly a spaceship along the bottom of the screen, catch falling aliens, and dodge rocks. (I wrote about building it in What I Learned Building My First Game.) Higher levels get crowded fast: meteor storms, UFOs, gravity wells that drag you off course, “scramble” effects that mess with your controls, and bonus aliens that drop hearts, shields or bombs. There's also a coin shop where you buy bullets, shields and extra lives in the middle of a run.

After playing it for a while I wanted to know if a machine could learn it. In rl-aliencatcher the ship is controlled by an agent trained with PPO (Proximal Policy Optimization), playing with every rule and hazard switched on.

Don't Rebuild the Game, Use It

RL game projects usually start by rewriting the game in Python so it plugs into Gym-style tooling. I didn't want that, because the copy always drifts. You change a spawn rate or a hitbox in one version and forget the other, and then the agent gets very good at a game nobody plays.

So I set one rule: the unchanged game is the training environment. The game's main.js is copied byte-for-byte into the training harness. During training it runs headless in Node.js, and when you watch, the same file runs in the browser. Training and playback can't diverge because there's only one game.

training:
  train.py (stable-baselines3 PPO)
     └─▶ alien_catcher_env.py (Gymnasium env)
           └─ stdio JSON ─▶ env_server.js
                              └─▶ headless.js + shim.js (fake DOM, fake clock)
                                    └─▶ game/main.js   ← the real, unmodified game

playback:
  index.html ─▶ main.js (real game, real rendering)
                ai.js ─▶ policy.js (tiny MLP forward pass) ─▶ injects key presses

Running a Browser Game Without a Browser

The game expects window, document, a <canvas>, Image, AudioContext, requestAnimationFrame and timers. Node has none of them, so I wrote bridge/shim.js, a small fake browser.

• A canvas context that does nothing. It's built on a JavaScript Proxy: any drawing call (fillRect, arc, gradients) returns an empty function, and any style assignment is accepted and thrown away. Rendering costs zero.
• Fake DOM elements with a working classList, event listeners and .click(), so the harness presses “Start” and “Restart” the way a person would.
• A clock I control. This mattered more than anything else in the shim, so it gets its own section.

The fake clock

Several mechanics in the game run on time: invulnerability frames after a hit, level-up banners, reset timers. On real time, training would be slow and nondeterministic. The shim swaps requestAnimationFrame, setTimeout and setInterval for one scheduler. Each call to pumpFrame() moves it forward exactly 1/60th of a second. Timers that are due fire first, then the animation callbacks run.

The game believes it's running at 60fps in a browser. Really it runs as fast as the CPU allows, and the same inputs always give the same result. On my laptop that came out to about 1,100–1,400 environment steps per second across 8 parallel Node processes.

On top of the shim, headless.js exposes the three functions an RL environment needs: reset(), step(action) and getState(). Each agent step applies an action and then advances 4 game frames (frame skip), so the agent decides about 15 times per second.

What the Agent Sees, Does and Wants

Observations: 100 numbers

The agent doesn't look at pixels. It gets a normalized vector of 100 floats, built in engine/obs.js:

• Ship and metadata (9 values): x position, lives, coins, bullets, shields, level, plus flags for an active shield, scramble and meteor storm.
• The 6 nearest rocks: relative position, fall speed, width, and whether it's a boss rock.
• The 6 nearest aliens: relative position, speed, whether one is sitting on a rock, and whether it's golden or a queen.
• Everything else: the 4 nearest meteors, 3 bonus aliens (heart, shield or bomb), the UFO, the gravity well and the Last Egg, which is the win condition.

“Nearest” means lowest on the screen, since that's the most urgent. Positions are relative to the ship, so the policy learns “rock slightly to my left and close” instead of memorizing screen coordinates.

The detail I care about most: obs.js and the state reader readState.js are shared between the Node trainer and the browser. The export script copies them into web/engine/, so the policy gets exactly the same input in both places.

Actions: the full human control set

The action space is MultiDiscrete([3, 2, 2, 4]):

• move: left, stay or right
• shoot: yes or no
• shield: yes or no
• buy: nothing, bullets, shield or life

The agent “presses keys” by dispatching synthetic keyboard events to the game, exactly like a player. It gets no back door into the game state.

Reward

+ score gained (each catch)
+ 0.02 × coins earned
− 5     per life lost
+ 1     per life gained
+ 0.01  per step survived
− small penalty for being horizontally far from the nearest catchable alien
− 10    on game over
+ 50    for reaching the Last Egg (winning)

The distance penalty is there to give a brand-new policy something to go on. Without it, a fresh agent almost never catches an alien by chance and gets very little useful signal. If you've read my reward hacking series, you know shaping terms like this are exactly where agents find loopholes, so I kept it small.

Hard Rules: When the Agent Shouldn't Decide

One of the two commits in the repo is called shield fix. In the game, the shield only deflects meteor-storm rocks. An untrained agent has no idea, so it burned shields at random and had none left when a storm arrived.

I could have tried to teach that through reward shaping. Instead I wrote a small maskAction() function that both training and playback apply:

• Shields can only be activated during a meteor storm.
• Restock automatically: when shields drop to 1 or fewer and there are at least 70 coins, buy one.

Not everything has to be learned. If a rule is obviously right and the agent would need millions of steps to stumble onto it, write the rule and let the network spend its capacity on the hard stuff, like dodging and positioning. The mask lives in the shared engine, so the browser agent plays by the same rules it trained under.

Training

The setup is standard stable-baselines3 PPO:

I did three main runs, saving a checkpoint every 20k steps. Here's what the logs show:

• Episode length went from about 590 steps at the start of the first run to about 1,950 by the end of the latest one. At 4 frames per step, that's surviving roughly 40 seconds at first and over 2 minutes later, in a game that keeps getting harder.
• Mean episode reward went from −24.5 to −4.2.

Yes, reward is still negative on average. With −10 for game over and −5 per lost life, any episode that ends in death starts deep in the red. The agent hasn't beaten the game and doesn't reliably reach the Last Egg. What it has learned is to survive much longer and catch much more, and the curves were still climbing when each run ended. The action space is big, there are a lot of hazards, and more steps keep helping.

Shipping the Brain to the Browser Without an ML Library

Pulling TensorFlow.js or ONNX into the page for a 100 → 256 → 256 → 11 network felt silly. export_policy.py reads the trained PyTorch weights and writes them to a plain policy.json (about 2 MB). Then web/policy.js runs the forward pass in about 40 lines of JavaScript: matrix multiply, tanh, matrix multiply, and an argmax over each action group.

A parity test, bridge/test_policy.js, checks that the JavaScript output matches stable-baselines3 exactly. Hand-written inference is where a transposed matrix or a wrong activation slips in. Nothing crashes. You just get an agent that plays a bit worse and no idea why.

In the browser, ai.js:

• starts the game on its own,
• every 4 frames reads the live state, builds the same observation, runs the policy, applies the same action mask and presses the keys,
• shows a small badge with what it's doing (◀ left · 🔫 shoot · score 340 · lvl 4 · ❤ 2),
• and restarts after game over to keep playing.

Testing Something That Spans Two Languages

With Node, Python and the browser all involved, each layer gets its own test:

node bridge/test_random.js        # full games run headlessly with random actions
node bridge/test_policy.js        # browser policy.js == SB3 output (parity)
node web/test_browser.mjs         # headless Chromium: AI auto-plays, no console errors
python train/alien_catcher_env.py # Python <-> Node bridge smoke test

When something breaks, the failing test tells me which side of the bridge to look at, which saves a lot of guessing.

What I'd Reuse on the Next Project

The real game as the environment. A browser shim with a controllable clock is a small amount of code, and it gets rid of every “training and playback disagree” bug before it exists.

Writing the obvious rules down. The shield mask took a few lines. Teaching the same thing through reward would have taken a long time and probably still leaked.

Skipping the framework for small policies. A JSON file of weights and a short forward pass are enough, as long as a parity test is watching them.

As for the agent itself, it's clearly learning and clearly not done. Next up are longer runs, curriculum learning that starts at easier levels, and maybe a short history of frames in the observation so it can judge how fast things are moving.