Color Bump 3D — Computer Vision Controls

Overview

Color Bump 3D reimagines a classic mobile game as a multimodal HCI experiment. The game runs in Unity, but its input is driven entirely by a Python backend using OpenCV and audio analysis — no keyboard, mouse, or gamepad involved. Two players share the same physical space: one controls direction with their head, the other controls speed with their hands and voice.

Player 1 — Head Orientation Tracking

• Webcam feed processed in real-time by Python + OpenCV to detect the player's face
• Facial landmark detection used to estimate head pitch and yaw angles
• Head orientation mapped to ball steering direction — tilt left/right to steer, tilt forward/back to adjust
• Angle thresholds and dead zones configured to avoid jitter and accidental input
• Control data sent to Unity via a local socket connection at ~30Hz

Player 2 — Hand Gesture & Audio Control

• Hand gesture recognition via OpenCV contour analysis — detects open/closed hand and finger count
• Number of fingers extended maps to ball speed tier (0 fingers = slowest, 5 = fastest)
• Audio input analyzed for volume level — loud sounds trigger a speed burst mechanic
• Both gesture and audio signals combined and sent to Unity through the same socket channel

Unity — Game & Integration Layer

• Unity C# socket client reads the Python control stream each frame and applies it to the ball's Rigidbody
• Color Bump gameplay: ball rolls along a track, colliding with same-color obstacles passes through, wrong-color obstacles end the run
• Multiplayer session management — two control channels handled simultaneously
• Visual feedback shows both players' current input state on-screen in real time

What I Learned

Bridging two separate runtimes (Python and Unity) in real-time was the biggest technical challenge. Latency in the socket communication had to stay under ~50ms to feel responsive — this required careful buffering on the Python side and interpolation on the Unity side. The project also pushed me to think about input design from a HCI perspective: how do you make a non-standard control scheme feel intuitive within 30 seconds?