Listening for laughter on construction sites

This summer I was a software engineering intern at FactorLab in Pleasanton. My project was to build a system that listens to recordings from construction sites and finds the moments when people laugh, as one signal for worker-safety insights.

That sounds simple. People laugh, you find the laughing. It turned out that construction sites are loud, and a lot of loud things sound a little like “ha ha ha.”

Cleaning up the audio

The recordings come from the field, so they’re noisy and multilingual, and a lot of each one is dead air. Before any model could look at them, I built a cleaning pipeline. It cuts out the long stretches of silence, reduces background noise, brings every file to the same loudness, and resamples everything to the same format.

That last step mattered more than I expected. Later on, my laughter clips and my non-laughter clips came from different sources at different sample rates. A model could have learned to tell them apart from the sample rate alone, without ever listening for laughter. It would have scored great and learned nothing. So everything gets resampled the same way before a model sees it.

Picking a model

I didn’t train a model from scratch. I started by testing three pretrained ones on a few labeled videos: Google’s YAMNet, PANNs, and a laughter detector from Gillick et al. YAMNet and PANNs are general sound classifiers, and to catch laughter at all I had to drop their confidence threshold to 0.015. Gillick’s model is built specifically for laughter, and it found every burst in two of the three videos.

None of them was right every time, so I combined two. The pipeline blends Gillick’s model with YAMNet, weighted 60/40, and uses YAMNet to veto the usual impostors: coughs, sneezes, thuds, bangs, grunts, and cheering. Then I ran it across all 1.6 million files in the company’s audio database and used its most confident detections, plus a lot of hard negatives, to build a training and test set.

Teaching it what laughter isn’t

The hardest part wasn’t finding laughter. It was everything the model thought was laughter. Coughing was the worst offender. A cough is short, sharp, and sometimes comes in bursts, which is close enough to a laugh to fool a model. Cheering and crowd noise caused problems too.

So I went looking for exactly those sounds. I searched back through the company’s audio files for more examples of coughing and the other sounds the model kept getting wrong, and added them to the training data as negatives. Each round, the model saw more of the things it confused with laughter, and learned to tell them apart.

Then I fine-tuned Gillick’s model on that data. Since it was already built for laughter, it had less to learn. I froze its early layers and trained only the later ones, so a few thousand clips wouldn’t wipe out what it already knew. The fine-tuned model reached 94% accuracy.

Along the way, I presented my progress to FactorLab’s CTO and CEO. The feature is now being shipped to FactorLab’s platform, which is a strange and good feeling for something I started on in June.