In the whisper code, they set their ratio threshold to be 2.4. You can see an example of this in the Non-speech section below. The goal of this threshold is to prevent the model from outputting predictions where it has gotten stuck and generated the same phrase over and over. Zlib’s compression capitalizes on repeated sequences, so a text string with many repeated sequences will be able to compress down more than a string with more unique sequences. This is because they repeatedly run inference with different decoding strategies until they meet the heuristics.Ĭompression ratio: the compression ratio heuristic is defined by the calculation:Ĭompression_ratio = len(text)/len(press(text.encode("utf-8"))) The way these strategies are currently implemented in their code results in slightly improved test results, but can slow down inference by up to 6x. In the Whisper paper, they describe their complex decoding strategy including a few heuristics they landed on to try and make transcription more reliable. transcription = anscribe("hello_world.mp3", task=”transcribe”, language="en") You can call the transcribe function without explicitly setting the decode options and it will set some defaults for you. In order to run the transcribe function, you need to make some decisions about how you want to decode the model's predictions into text. Whisper makes it very easy to transcribe an audio file from its path on the file system and will take care of loading the audio using ffmpeg and featurizing it before running inference on the file. The models’ parameter sizes range from 39 M for tiny and up to 1550 M for large. Whisper is an encoder-decoder transformer model that takes in audio features and generates text. Large (the large model is only available in the multi-language form) Whisper is available as multilingual models, but we will focus on the english only versions here. Model = whisper.load_model("medium.en", device="cuda") Then, you can import Whisper in your own python code and use their load_model function to download the pre-trained weights and initialize one of the models. To install Whisper on your machine, you can follow the Setup guide in their readme which walks you through the steps. You can run their command line tool, which will set a bunch of parameters for you, but you can also play around with those parameters and change what kind of results you get. When experimenting with Whisper, you have a few options. In this blog, we will explore some of the options in Whisper’s inference and see how they impact results. OpenAI recently released a new open source ASR model named Whisper and a repo full of tools that make it easy to try it out.
0 Comments
Leave a Reply. |