What is Whisper, the speech recognition model?
OpenAI's open-source speech recognition model explained: what it does, how it approaches transcription, and where Clipy's own transcription differs.
What Whisper is
Whisper is a speech recognition model released by OpenAI as open source. It was trained on a large amount of multilingual audio and approaches transcription as a single learning problem: given audio, produce the text, with language identification and timestamps as part of the task.
Because the model and its weights are public, it can run locally on suitable hardware, and a wide ecosystem of transcription tools has been built on top of it.
Reviewing what to fix in the transcript
Whisper handles many languages and accents and copes with varied audio conditions, which comes from the breadth of its training data. It remains a prediction system: names, numbers and code-switched speech are where transcripts tend to need review, timing estimates can drift against real audio, and sung or overlapping speech is harder than a single clear voice.
Treat any Whisper transcript as a first draft worth a read-through, not a finished transcript.
How this relates to Clipy
Clipy currently uses Deepgram for transcription, not Whisper: audio is extracted from your file locally as WAV and sent to Deepgram, with the video itself staying on your device. The practical implications are similar either way. Generate the transcript, review it against the audio, fix names and timing, then export subtitles or burn them in.
Common questions
Is Whisper free?
The model and code are open source and can be run without paying for the model itself, but running it well takes suitable hardware, and hosted services built on it may charge.
Does Whisper translate?
The model includes a task that translates speech into English as part of its training. Clipy's own translation flow works differently: it transcribes first, then sends the subtitle text to Gemini for translation.
Further reading
Try the workflow
Prepare your own file and review the result before using it.
Open the related Clipy tool