writing a scripting VM in rust

I'm building a game in Unreal, and gameplay logic in C++ means a rebuild every time I change a number. That loop is slow enough that I stop experimenting, which is the worst thing a tool can do to you.

The fix is scripts. The excuse is that I wanted to learn Rust properly, so I'm writing the whole thing myself: a small language, a compiler, and a VM to run the bytecode.

it started in C++#

The first version was C++, which made sense at the time — the game is already C++, so the VM could just reach into game state directly. It worked. It was also the kind of code where a crash three frames later was caused by something I did three functions ago.

Rewriting it in Rust was supposed to be a learning exercise. It turned into the version I actually use.

stack-based, for now#

It's a stack VM. Every tutorial starts with a stack VM, and after the port I never had a good enough reason to move off it — the interpreter is small, the compiler is almost trivially simple, and none of my scripts are hot enough for dispatch count to be the thing that hurts.

enum Op {
    LoadConst(u16),
    Add,
    JumpIfFalse(u32),
    Call { func: u16, argc: u8 },
    Return,
}

The cost is obvious when you read the bytecode: push, push, add, store for what is conceptually one operation. Registers are where I want to take it — wider instructions, far fewer dispatches per line of script — but that means writing a register allocator, and I'd rather have hot reload working first.

what rust actually taught me#

The borrow checker is the reason this works. The C++ version did the obvious thing: hand the VM a pointer to the host game state and let it write wherever. Rust refused to let me port that straight across, so I ended up with an explicit host interface — the script asks, the host answers, nothing reaches across.

That boundary is the good part of the design and I didn't come up with it. The compiler forced it on me.

still missing#

• Garbage collection. Strings currently live forever. I'm going to pretend that's a feature until it isn't.
• Debug info. A runtime error gives you a bytecode offset, which is useless. Line numbers next.
• Hot reload, the whole reason for the project. Swapping a function's bytecode mid-frame without invalidating live call frames is the interesting problem, and I haven't solved it yet.

Gets rewritten roughly every other month. That's fine — the rewrites are where the learning is.