Procedural Terrain Generation via Neural Style Transfer + Genetic Algorithms

The Problem

3D terrain generation for games typically relies on hand-crafted noise functions or artist-painted heightmaps. Neither scales well — noise lacks visual variety and artistic intent, while manual authoring doesn't scale to procedurally generated worlds. The goal of this thesis was to build a system that generates terrain that is both visually compelling (aesthetics from a reference image) and functionally playable (navigable by AI agents).

The Approach

The pipeline has three main stages:

• Neural Style Transfer (NST): A pretrained VGG convolutional network extracts perceptual features from a style reference image. These features guide the terrain's visual character — rocky, mountainous, flat, eroded, etc.
• Genetic Algorithm: A population of terrain candidates evolves over generations using a multi-objective fitness function. Elitism ensures the best solutions survive. Adaptive clamping and early stopping prevent overfitting.
• NavMesh Fitness Metric: The key novel contribution. Each terrain candidate is baked into Unity's NavMesh at runtime and evaluated using a custom connectivity metric — measuring what fraction of the terrain surface is reachable by an AI agent. This makes "playability" a first-class optimization target.

Results

The system was validated across four terrain archetypes: Plateau, Ridge, River, and Plain. Each archetype was generated from a distinct style reference, and the NavMesh metric confirmed that all outputs maintained acceptable agent traversability while adopting the visual character of the reference image.

• Successfully generated diverse terrain styles without manual heightmap authoring
• NavMesh connectivity metric proved effective as a real-time playability signal
• Multi-criteria fitness (SSIM + perceptual NST loss + navigability) produced stable convergence
• Pipeline runs on GPU — generation of a full terrain takes seconds, not minutes

What Was Hard

Balancing visual fidelity and navigability was the central challenge. A terrain that looks exactly like a reference image might be a completely impassable cliff — and a perfectly flat, walkable terrain has no visual character. The multi-criteria fitness function required careful tuning of weights between the SSIM metric, the NST perceptual loss, and the NavMesh score. Early stopping was critical to prevent the genetic algorithm from collapsing into local optima that maximized one signal at the cost of others.

The real-time NavMesh baking in Unity was also non-trivial — standard baking is an async editor-time operation and had to be restructured for synchronous runtime evaluation inside the genetic algorithm loop.