EXPLORE

Gesture Gaming: spatial interaction & ultrasonic arrays.

A hybrid physical-digital gaming system uniting 21-point MediaPipe hand landmark tracking with high-frequency ESP32 ultrasonic transducer arrays to bridge camera vision with physical proximity telemetry.

View GitHub Repository ↗
  • MediaPipe Vision
  • ESP32 Hardware
  • Serial Bus (115200 baud)
  • Sub-16ms Latency

The System.

Traditional camera-based gesture recognition struggles with depth ambiguity when hands move along the optical z-axis. To solve this, Gesture Gaming introduces an active multi-modal sensor fusion pipeline. A web camera captures high-resolution 2D palm and finger coordinates via a lightweight neural pipeline, while a custom ESP32 ultrasonic transducer array samples high-frequency distance metrics at 40 kHz.

The dual data streams are harmonized through an asynchronous Python backend, interpolating optical hand vectors with physical time-of-flight measurements to provide tactile precision without wearable sensors.

Gesture Gaming hardware bench and optical vision tracker display

The Bottleneck, The Solution.

Bottleneck

Z-Axis Depth Blindness in Monocular Vision

Standard webcam setups fail to compute exact physical distance when hands retreat toward the body, causing erratic false-triggering in fast-paced arcade interaction.

Breakthrough

Hardware-Optical Time-of-Flight Fusion

By mounting an ultrasonic transceiver array directly beneath the primary optical field of view, the system fuses physical sound reflection curves with MediaPipe 21-point skeletons at 60fps.

Under The Hood.

MediaPipe 0.10 OpenCV 4.8 ESP32-WROOM-32 HC-SR04 Transducer Array PySerial (115200 Baud) Pygame Physics Engine
0
Pipeline FPS
0
Inference Latency (ms)
0
Landmark Vectors
0
Reliability Score
Next project: Mnemosyne
Next Research Project ↗

Mnemosyne: academic synthesis
& exam intelligence.