Gesture Gaming: spatial interaction & ultrasonic arrays.
A hybrid physical-digital gaming system uniting 21-point MediaPipe hand landmark tracking with high-frequency ESP32 ultrasonic transducer arrays to bridge camera vision with physical proximity telemetry.
The System.
Traditional camera-based gesture recognition struggles with depth ambiguity when hands move along the optical z-axis. To solve this, Gesture Gaming introduces an active multi-modal sensor fusion pipeline. A web camera captures high-resolution 2D palm and finger coordinates via a lightweight neural pipeline, while a custom ESP32 ultrasonic transducer array samples high-frequency distance metrics at 40 kHz.
The dual data streams are harmonized through an asynchronous Python backend, interpolating optical hand vectors with physical time-of-flight measurements to provide tactile precision without wearable sensors.
The Bottleneck, The Solution.
Z-Axis Depth Blindness in Monocular Vision
Standard webcam setups fail to compute exact physical distance when hands retreat toward the body, causing erratic false-triggering in fast-paced arcade interaction.
Hardware-Optical Time-of-Flight Fusion
By mounting an ultrasonic transceiver array directly beneath the primary optical field of view, the system fuses physical sound reflection curves with MediaPipe 21-point skeletons at 60fps.