Turn a pinch into a pointer

The problem

A projector can show a workspace, but interacting with it usually sends you back to a mouse. Arc Wall uses a webcam to track hands and a canvas to draw the scene. The laptop remains the control desk, with a separate projector window.

How it works

1. Find the hand's landmarks

   MediaPipe estimates hand landmarks, points such as fingertips and knuckles. The gesture interpreter mirrors those coordinates and smooths the index fingertip position to reduce pointer jitter.

2. Interpret a pinch

   Thumb-to-index distance is compared with palm size. Different engage and release thresholds help prevent flickering between pinched and open. The same pointer can grab objects, draw, or erase; two open hands can resize a selected object.

3. Fit and share the canvas

   Four corner handles define a perspective transform, a mapping from a rectangle to a wall shape. The code rejects crossed or collapsed corners. The control and output windows share scene changes through BroadcastChannel, a browser messaging channel.

Decisions & tradeoffs

Try it yourself

From an Arcwall checkout, with Node.js 20.9+ and npm, start the local studio. No AI configuration is needed for drawing or local commands.

npm ci
npm run dev

Open http://127.0.0.1:8787. Draw with the mouse, try Undo, then try “add a circle.” Hand tracking needs camera permission. Use Launch projector only when you want the second window; drawing also works on the laptop.