A series · started October 2026
Learning World Models
These notes follow Build a World Model from Scratch by Dr. Rajat Dandekar, which is where the MiniPong build and the renderer / simulator / planner map come from. The original paper is Ha & Schmidhuber’s World Models (2018).
Every time I read about world models I found a different definition. Some people meant video generators, some meant reinforcement learning, some meant Yann LeCun’s JEPA. So I decided to learn the topic properly, from the beginning, and to build the smallest working version myself. This series is what I wrote down along the way.
Here is the one sentence I kept coming back to:
A world model is a learned stand-in for the environment. Give it what I see now and what I’m about to do, and it tells me what I’ll see next.
Learn that once from recorded play, and you can practise inside it as often as you like, without touching the real world. Everything below unpacks that sentence.
All the code lives in one repository, learning-world-models, with each example small enough to read in one sitting and run on a laptop CPU.
Roadmap
- 1Published · Oct 2026 · 12 min read
What a world model is, and the tools to build one
The agent–environment loop, the renderer / simulator / planner map, state versus observation, and the four tools every paper assumes: compression, return, value and actor–critic.
- What a world model actually is
- The toolkit: compression, return, value
- 2Published · Oct 2026 · 14 min read
MiniPong: building V and M from scratch
Squeezing a 32×32 frame into 12 numbers, the four small decisions that make it work, a memory that works out velocity on its own, and why the dream drifts.
- Building V: squeezing a frame into 12 numbers
- Building M, and why dreams drift
- 3Published · Oct 2026 · 18 min read
Learning inside a dream, and what breaks without the tricks
A controller trained by evolution, how agents cheat a flawed dream, and my experiment removing each MiniPong trick one at a time. Plus a short appendix on the VAE.
- C: learning to act inside a dream
- My experiment: what breaks without the tricks
- Appendix: the VAE and the reparameterisation trick
- 4Planned
Dreamer: learning behaviour from imagination
An actor–critic trained entirely on imagined rollouts in a learned latent space.
- 5Planned
Tokens and transformers as world models
IRIS: turning frames into discrete tokens and predicting them like language.
- 6Planned
Diffusion world models
DIAMOND: keeping long dreams sharp by generating each frame with diffusion.
- 7Planned
Predicting without pixels
JEPA: learning a world model that never has to redraw a single frame.