A hand gesture becomes a mouse click
A camera loop, finger geometry, and direct desktop actions.
A mouse controlled by hand gestures.
Python / OpenCV
The problem
GestureFlow explores controlling a desktop mouse with a webcam instead of a physical mouse. The virtual_mouse repository contains a small Python program using OpenCV for camera frames, MediaPipe for hand landmarks, and desktop input libraries for actions.
How it works
- Camera frameOne hand
- Angle rulesChoose gesture
- Desktop actionMove or click
Read and mirror a frame
OpenCV reads from the default camera. Each frame is mirrored and converted to RGB before hand tracking. The tracker is configured for one hand, and the program collects its landmark coordinates.
Compare finger angles and distances
Helper functions calculate angles at finger joints and a scaled distance between hand points. Fixed threshold rules distinguish pointer movement, left click, right click, double click, and a screenshot gesture.
Act on the desktop
The program maps the index fingertip to screen coordinates and sends mouse actions through PyAutoGUI and pynput. The screenshot gesture writes a PNG in the working directory. The camera window displays the detected action, and q exits the loop.
Decisions & tradeoffs
Rules are easy to inspect
A few thresholds make it clear why a gesture fires. Camera angle and hand posture can still affect the result. The loop can send an action on successive matching frames; the code does not establish a measured false-click rate.
The actions are real
This program can move the system pointer, click, and save screenshots. The repository has no README or pinned installation recipe, so a source walkthrough is the most accurate thing to offer here.
Try it yourself
Open main.py and util.py through the links below. Start at identify_gesture and follow each branch to its desktop action. Reading the code does not activate a camera or move the pointer.
Compare the left-click and right-click angle tests. Then find where q exits and where screenshots are saved. Those details matter before choosing to run any desktop-control program on your own machine.
Sources & scope
Limited coverage: the public repository has two Python source files and no README or dependency lock. No local checkout was available, and no accuracy or portability result is claimed.