We measured how many ums speech-to-text keeps
Most transcript-based editors offer to remove filler words. Select the ums, one click, gone. What the demo never mentions is that the feature can only cut the fillers the transcript wrote down - and whether they get written down at all depends on the speech-to-text engine, and on how it was run.
So we measured it.
Synthetic speech can't answer this
Our first attempts used generated test audio, and they told us nothing: macOS
say pronounces "um" as an ordinary word, the way nobody hesitates. A test of
filler words needs people actually hesitating.
We used ten clips of real, spontaneous meeting speech from the AMI Meeting Corpus - 11.8 minutes, 1,513 words, 184 fillers - because its human transcripts are verbatim, with the speakers' ums annotated. Three engines, all running on the Mac, measured on a MacBook Pro M4 with 16 GB:
- Parakeet TDT 0.6B v3, NVIDIA's open speech model
- Apple's SpeechAnalyzer, the speech API built into macOS 26
- whisper.cpp with the large-v3-turbo model, on whisper.cpp's default settings - the way Rooster runs it as its fallback engine
The results
How many of the fillers the speakers actually said each engine kept:
| Engine | Fillers kept | Word error rate | Speed |
|---|---|---|---|
| Parakeet | 79.3 % (146 of 184) | 10.6 % | 306.8x realtime |
| SpeechAnalyzer | 31.5 % (58 of 184) | 18.1 % | 99.9x realtime |
| whisper.cpp | 5.4 % (10 of 184) | 20.0 % | 21.0x realtime |
All ten of whisper.cpp's fillers came from one clip. On the other nine it wrote down none at all.
One caveat we checked rather than assumed: AMI is in Parakeet's training data, and eight of these ten clips come from the part it trained on. So we ran ten more clips from meetings outside that part. Parakeet kept 83.6 % of the fillers there - better, not worse - SpeechAnalyzer 44.5 % and whisper.cpp none at all. The fair summary is that Parakeet keeps about four in five, SpeechAnalyzer between a third and a half, and whisper.cpp close to none.
The striking part is what happens when you score only the words that carry meaning - take the fillers out of both the reference and the transcript. The three engines land within 2.3 points of each other. This isn't a difference in how well they hear. It's a difference in what they choose to write down.
It's a setting, not the model
To prove that, we gave whisper.cpp one generic initial_prompt asking for a
verbatim style - no words from the clips, nothing else changed. Its filler
recall went from 5.4 % to 88.6 %, past Parakeet, and its word error rate fell
from 20.0 % to 9.5 %.
Then we tested the prompt more widely, on scripted test audio in five other languages and a 24-minute file - generated speech again, which is fine for these questions because the script is its own answer key. We didn't ship the prompt:
- On one of two French clips, it translated instead of transcribing. The same audio that whisper.cpp transcribes correctly in French came back in English, with the engine still reporting French. Whisper.cpp is in Rooster precisely for the languages Parakeet doesn't cover, so this is the case that matters most.
- On the 24-minute file, it deleted speech. About 66 words in two places, including a whole 15-second stretch that every other configuration transcribed, were simply missing - and a transcript with a hole in it still reads perfectly well.
A transcript that quietly loses a sentence is worse than one that leaves out the ums.
What we did about it
Parakeet is Rooster's default English engine. It writes fillers down without being asked, it was the fastest of the three by a wide margin, and it peaked at 136 MB of memory against whisper.cpp's 1.9 GB.
Every transcript Rooster makes records which engine made it, and how. An empty filler list means something completely different coming from Parakeet than from an engine set up not to capture fillers, and Rooster couldn't tell the two apart until it kept that record.
When it finds no fillers, Rooster says why. On a Parakeet transcript it says the engine writes fillers down, so most likely none were said. On a transcript from an engine configured not to capture them, it says so and tells you to switch to Parakeet and transcribe again. A feature that silently returns nothing looks broken; one that explains itself doesn't.
What it means if you're choosing a tool
- Test filler removal on your own recording, with hesitations you know are there. A list of zero isn't proof that you didn't say any.
- If you build on speech-to-text, measure the configuration you ship, not the model's name. The same Whisper weights scored 5 % and 89 % here.
- Detecting the ums is half the job. Cutting them without flattening the delivery is the other half.
All three engines ran on a laptop, with nothing uploaded and nothing metered. Transcribing a recording and finding its ums is exactly the kind of everyday AI job your Mac can already do. How Rooster's filler removal works, step by step, is in Remove filler words.