Archit Sunat All work

Controlling music with hand gestures

Condukt

A macOS menu bar app that watches your webcam and turns hand gestures into media controls, built in Python on Google's MediaPipe. The first thing I have taken end to end on my own, from the idea to something I use every day.

Play, two skips, volume down, volume up, pause. Sound on: the volume slides as he holds the gesture.
01

The idea

Change the track without touching anything.

I built this end to end with Claude Code to find out first hand what building that way changes. Something small enough to finish, and all of it mine to decide: the idea, the recognition, the tuning, the packaging and the permissions.

Closed fist pauses. Open palm plays, and swiped sideways it skips. Thumbs up and down move the volume. Eleven mappings over seven hand shapes, because the same open palm means play when it is still and skip when it moves.

The whole interface. A menu bar dropdown and its settings, with no window and no view of what the camera can see.
The whole interface. A menu bar dropdown and its settings, with no window and no view of what the camera can see.
02

The real problem

The model changes its mind, and the app has to decide when to ignore it.

Recognition comes pre-packed with Google's MediaPipe model: it watches one hand, names the shape and scores how sure it is, thirty times a second. Hold your hand perfectly still and the answer still changes. A frame gets dropped. Your hand drifts out of view. Every one of those looks exactly like you changing your mind, so a naive build fires four times when you meant once. Almost all of the work went into deciding when to ignore the model. None of it is machine learning; it is interaction design on top of a sensor that is often wrong.

One gesture from the hand appearing, through the 0.7 second hold, to the action firing and the 1.5 second lockout HOW SURE THE MODEL IS THAT IT SEES AN OPEN PALM 0 0.5S 1.0S 1.5S 2.0S 2.5S 0.5 1.0 BELOW HERE THE FRAME COUNTS AS NOTHING PALM GOES UP THE TRACK PLAYS TIMER COUNTING  ·  0.7S WILL NOT FIRE AGAIN  ·  1.5S Palm up and held still. Once the model is sure enough, a timer starts. At 0.7 seconds the track plays. For the next 1.5 seconds the same palm does nothing at all, so one gesture is one action rather than twenty.

  1. Hold

    A shape has to hold for 0.7 seconds before anything happens. Act on single frames and the app fires at everything your hands do.

  2. Grace

    If the model loses the shape for under 150 milliseconds the timer keeps counting. A dropped frame is not a change of mind.

  3. Cooldown

    Once an action fires it cannot fire again for 1.5 seconds. Volume is the exception at 0.5, because there repeating is the point.

  4. Floor

    Under 50% sure and the frame counts as nothing. That is the model's own score for the gesture it picked, not a probability that it is right.

03

What I learned

Tuning the four numbers against an eval set instead of by feel.

Those numbers started as guesses, and every time I changed one I was back to waving at a laptop and forming an opinion. So I recorded myself using it and marked what I had meant at each moment, which is an eval set, then wrote a harness that replays those recordings, tries different numbers and keeps a change only when the score goes up. It works because the part that decides touches no camera and no music app, so it can run offline thousands of times.

The autotuner loop: perturb a parameter, replay a recorded session, score it, keep it only if it improved Session EVERY FRAME LABELLED Perturb ONE PARAMETER Replay OFFLINE, NO CAMERA Score TRUE VS FALSE FIRES KEEP IT ONLY IF THE SCORE WENT UP A question I had been answering by waving at a laptop became a number, and a change could be judged in seconds. SIX PARAMETERS IN THE SEARCH DEBOUNCE 0.2 to 1.5COOLDOWN 0.8 to 3.0GRACE 0.05 to 0.4SWIPE THRESHOLD 0.03 to 0.15SWIPE WINDOW 0.3 to 1.5SWIPE COOLDOWN 0.5 to 3.0 The confidence floor is deliberately left out of it. 0.5 was already right, and searching it only rejects real gestures.

Design for a machine that is unsure

The model hands you a guess and a number. Everything good here is in how it treats that guess being wrong, not in the recognition.

Turn taste into a measurement where you can

Four numbers I had been arguing about with myself became a score over a recorded session. Most product questions cannot be handled that way. This one could.

The only way to review it is to use it

Nothing can tell you whether a thing feels right, so reading the code back tells you almost nothing. Most of what I changed came from running it every day.

I will not write "make it feel responsive" into a spec again without knowing that here it meant four numbers and a harness built to find them.

About

Most of the seven years was spent alongside the CEO, the head of design, engineering, business analysts, subject matter experts and growth.

What I want next is AI-native product work, and a place where I can keep building great products myself.

Education

  • MS, Cognitive Science

    Indian Institute of Technology Gandhinagar2015 to 2017

    Thesis on the cognitive system of musical improvisation, read through phenomenology.

  • BS, Life Science and Biochemistry

    St. Xavier's College, Mumbai2012 to 2015