From online meetings to voice memos and media content, the amount of audio data generated by companies daily is as vast as it is valuable.
However, listening to hours of audio content to extract key information is impractical. To enhance productivity and optimize workflows, the capability to quickly generate brief summaries from audio data is crucial.
Luckily, recent advancements in the field of automatic speech recognition (ASR) and natural language processing (NLP) have become robust and accessible enough to make custom audio summarization possible for a range of projects.
In this step-by-step guide, we explain how to summarize audio using OpenAI's Whisper ASR for transcription and GPT 3.5 to generate prompt-based summaries. We will explain how the different components work, and, most importantly, show you how to build your summarization API. We will also present Gladia as an enterprise-grade alternative to the Whisper API for transcription and demonstrate how to use it.
Speech recognition is the process of training an AI system to efficiently process spoken words in any language and translate them into a readable text format. Today, speech recognition systems such as Whisper ASR by OpenAI are trained using a sequence-to-sequence learning approach which consists of an encoder and a decoder block.
More specifically, a standard automatic speech recognition (ASR) system today works by receiving audio input from any recorded audio file, after which the digital audio signals are processed to extract acoustic features (spectrograms) relevant to the speech recognition process. The resulting spectrograms from the audio signals are then passed into an acoustic model (encoder) which is responsible for mapping them to a sequence of words which is usually a direct translation of the input audio file.
The encoder used in ASR is usually implemented using deep learning methods such as Long Short Term Memory (LSTM) networks or Convolutional Neural Networks (CNN). The decoder in the ASR system is responsible for predicting the most probable translation using the output of the encoder. To learn more, have a look at our recent article on how speech-to-text systems work.
Whisper ASR was introduced as an open source model by OpenAI in 2022, and was made available as an API in 2023. The model was trained on 680,000 hours worth of labeled audio (audio and their corresponding translations) and can be utilized for tasks such as identifying the language spoken in an audio, transcription of an audio file into text, and translating an audio into another language.
With its transformer architecture based on sequence-to-sequence learning, is considered among the best open-source ASR engines ever build.
Gladia is powered by Whisper ASR, which was designed as a more optimized and feature-rich version of the model adapted to enterprise scale and needs.
Our API was developed to overcome the challenges and limitations of vanilla Whisper, such as hallucinations, long inference time, and usage limitations.
In this tutorial, we will demonstrate how to use both original Whisper API and Gladia’s enterprise version thereof to transcribe text and then summarize it using GPT-3.5.
Text summarization is the process of extracting and piecing together useful information from a long text to form a shorter text that retains the main discussion points or action items, while being easier to read and understand.
When summarizing a text, there are two major methods involved [1]
While a variety of LLMs are used to generate custom summaries automatically today, GPT 3.5 has emerged as one of the best tools to conduct both types of summarization.
GPT 3.5 is a large language model (LLM) developed by OpenAI, trained on billions of text examples. It can perform any task that involves generation, such as text generation, code generation, and text summarization. This model also can follow specific instructions given to it. For example, if you ask it to provide an abstractive summary of a meeting recorded in text format, it will follow those instructions.
As you can see, the model follows the instructions we fed into it and returns a generated text that is accurate in detail concerning the instruction. This process of specifying instructions to the GPT 3.5 model to generate text in the way you want is known as prompt engineering. This will be useful to us later in this tutorial when we build our application. Also, check our blog for more examples of summarization prompts for virtual meetings.
In this article, we will be building an audio summarization API and it is assumed that you already have a foundation in the Python programming language. First, we will learn how to transcribe audio using Whisper API and the alternative Gladia. Then we will learn how to summarize the transcript using the GPT-3 API after which we will learn how to build a unified endpoint for both transcription and summarization.
You will need the following to code along with the code examples shown in this tutorial:
1. An Integrated Development Environment (IDE) that supports Python e.g. VSCode
2. Python version - 3.11
3. Python libraries
If you have already tried to implement an audio summarization system similar to the one discussed in this tutorial using Whisper, GPT 3.5, and FastAPI, one problem you might have faced is a name attribute error that occurs when you try to upload the audio file using FastAPI’s UploadFile endpoint and pass it to the Whisper API. We will address this issue in the subsequent section below.
Note: Here's the GitHub Repository containing all the codes used in this tutorial.
OpenAI provides two ways to access the pre-trained Whisper model. The first is a free, open-source package you can access directly from GitHub which allows for loading the model directly onto your system. However, it is worth noting that the computational expenses of the open-source package increase with the size of the model being used which leads us in this tutorial to use the paid API alternative which is a more efficient solution.
First, we must download the required packages using the pip package manager. The requirements.txt file is available on GitHub.
Once the package is successfully downloaded, you need to create an API key from OpenAI’s website. This is an essential step to access the model and get charged appropriately per usage. Once you have created the API key, pass it into a variable that can be used in the code. Note that it is best practice to use a .env file to store your API keys but for the sake of this tutorial, we will use variables.
The next step is shown in the code sample below. We name this Python file main.py and we import the necessary libraries to be used and the API keys.
When trying to create endpoints using FastAPI, the first important step is to set up the server. To do this, you need to initialize an instance of the FastAPI class, which will be used to define the routes for each endpoint that is created.
We create an asynchronous function called ‘whisper’ with a route /whisper-transcribe/ and declare a parameter "file" with a UploadFile type. The reason we defined ‘UploadFile’ as the type of file parameter is to inform the app that the whisper function expects a file upload.
The next step involves reading the uploaded file into a BytesIO class. You may be wondering why we did this instead of only reading the file using await file.read(). The transcribe method from the Audio() object expects the file being passed into the file parameter to have a name attribute which await file.read() doesn’t provide. To pass the file name of the uploaded file into the name attribute of the BytesIO class, you can read the file into the object and assign the filename to the name attribute.
Then, you can create an instance of Audio and pass the model name, file, and API key as parameters into the transcribe method, as shown in the code example below.
Before testing out any new endpoints we create, make sure to add the following line of code at the last lines.
We can test out the new endpoint created by opening a terminal in the directory where the main.py file is located. Once the terminal is launched, run this code below
Proceed to edit the URL by adding /docs to open SwaggerUI which is an interface for testing endpoints.
To proceed, select the /whisper-transcribe/ endpoint to reveal more information. Then, select ‘Try it out’. This will reveal an option to upload a file, or an audio recording in this case of the tutorial.
From the response body, we can deduce that the Whisper API can accurately transcribe the text from the audio file we uploaded to test the endpoint created.
As an alternative to using open-source Whisper, Gladia provides a plug-and-play production-ready version of Whisper ASR, which can be used by simply using the requests library in Python.
Here’s a code sample on how to use Gladia API for transcription. The requests library is added to the main.py file to make request calls to Gladia’s API.
Next, we define a route named gladia using FastAPI and assign it to an asynchronous function defined with a parameter of type UploadFile. When using our API, you can either pass in an audio file or a URL to where the file is located.
In this tutorial, we will be passing in an audio file. To do this, our API expects three parameters: the name of the audio file, the audio file itself, and its content type. The UploadFile type provides access to the name using the .filename attribute and the content type using the .content_type attribute.
Subsequently, we define a header parameter using the API key, and a file parameter using the three attributes: filename, file, and content type.
Once everything needed to make a request is defined, we can send a request from which we will receive a JSON response. To retrieve the full transcription from the response, we need to loop through the response dictionary and retrieve each transcription.
To test the endpoint, simply run python -m main in your opened terminal.
The procedure for obtaining a response has been detailed above in ‘How to transcribe audio using Whisper ASR Webservice’.
In contrast to Whisper's API, which charges users for each request, Gladia offers a free plan that allows users to use our API for 10 hours per month without any restrictions on the many features we offer, such as batch transcription, speaker diarization, word-level timestamps, and live transcription.
To learn more about our free, pay-as-you-go and enterprise plans, please visit our pricing page.
For the audio summarization system developed in this tutorial, we will be making use of OpenAI’s GPT 3.5 API for summarization.
To use the GPT 3.5 API to summarize the transcripts produced by Whisper ASR, we define an endpoint named /summarize/ and define a Python function with a string parameter transcript that expects an audio transcript.
Earlier in the tutorial, we discussed prompt engineering and how it can be used to instruct the model on a desired output for a corresponding input text. In this code, we give the AI several instructions to summarize the transcript while accounting for and correcting mistakes in the transcript. We also made sure to instruct the model to avoid adding unnecessary information to its generated text.
Next, we initialize a method of the ChatCompletion class from the openai package. In this method, we define the GPT 3.5 model in the model parameter and pass a list containing a dictionary with the keys: role and content. The role is set to system to let the model know that the content being provided is a system-level instruction.
According to OpenAI ‘a system level instruction is used to guide your model's behavior throughout the conversation’. Learn more about the parameters available here.
Here, we use the same meeting text used in the example of GPT-3 above.
An automatic system will be defined as one that allows for audio upload using a FastAPI endpoint, passes the audio into Whisper for transcription, and immediately passes the transcription into GPT 3.5 for summarization which is returned to the end user.
To do this, you will be making little adjustments to the code in the whisper.py, gladia.py, and summarize.py files. The adjustments to be made include the conversion of the endpoints in each of the files into standalone Python functions.
The first step before making the adjustments involves moving all 3 files into an utils folder, while you also create a main.py file outside the utils folder.
Your file structure should look like this when done creating the file structure.
- whisper-summarizer
- main.py
- requirements.txt
The main folder, whisper-summarizer, should contain all subfolders and files. The requirements.txt file should look like the image below. The purpose of this file is to specify all the libraries or packages that need to be installed when working in a virtual local or cloud environment.
The main folder, whisper-summarizer, should contain all subfolders and files. The requirements.txt file should look like the image below. The purpose of this file is to specify all the libraries or packages that need to be installed when working in a virtual local or cloud environment.
To make our app fully functional, we would be adding standalone functions for each of the services being used e.g Gladia, Whisper, and GPT-3
The code samples we provided above show you how to convert the endpoints in the three files into standalone functions by removing the lines of the code that had to do with uploading and reading a file. This was done to create a unified endpoint that works for both the transcriber and summarizer and handles all forms of data input, such as audio upload or passing in a string.
To proceed with creating a unified endpoint, we will be updating the main.py file. As you have learned so far, we need to import the libraries we would be making use of as seen below.
The next step involves creating the server. A description is added to tell users about the purpose of the API and we have also titled the API as Whisper ASR Summarizer. We have also added a middleware using CORS for cross-communication between the Whisper ASR API and a frontend server.
The middleware accepts various parameters such as
1. Origin, which is used to specify any protocols and ports that can communicate the Whisper ASR API. Here we set it to “*” which means all origins are allowed to communicate with the backend.
2. Method, which is used to specify the HTTP methods that are allowed for communication with the backend. Here we set it to ‘POST’ and ‘GET’.
3. Header, which is used to specify the HTTP headers that are allowed for communication with the backend. Here we set it to Content-Type since we’ll be dealing with file uploads.
To transcribe using the Whisper ASR API, we first create an endpoint named /upload-audio-whisper/ and an asynchronous function. We then specify the file parameter to be of type UploadFile and add a description to explain what the parameter expects.
Next, we read the uploaded file, set its name with BytesIO, and pass it into the whisper_transcribe function to obtain a transcription. The transcript is then passed into the summarize_gpt function to obtain a summary, and both the transcript and summary are returned.
The steps for transcribing using Gladia are the same as above. First, we read the file. Then, we pass the uploaded file into the Gladia function. Next, we parse the output to retrieve the full sentences. Finally, we pass the resulting transcript into the GPT function to get a summary
You can view the full code here.
If you wish to deploy your Webservice, here is a tutorial on how to do so using Render.
In this tutorial, we have shown how Gladia and Whisper can be used to generate audio transcriptions, with GPT-3.5 used subsequently for different types of text generation, while leveraging FastAPI to build an API for audio summarization.
To learn about the many features Gladia provides for audio transcription, visit our developer documentation.
[1] Dutta, S., Das, A. K., Ghosh, S., & Samanta, D. (2022). Data analytics for social microblogging platforms. Elsevier.
At Gladia, we built an optimized version of Whisper in the form of an enterprise-grade API, adapted to real-life professional use cases and distinguished by exceptional accuracy, speed, extended multilingual capabilities and state-of-the-art features.
Speech-To-Text
Large Language Models (LLMs) are at the forefront of the democratization of AI and they continue to get more advanced. However, LLMs can suffer from performance issues, and produce inaccurate, misleading, or biased information, leading to poor user experience and creating difficulties for product builders.
Speech-To-Text
Over the last few years, Large Language Models (LLMs) have become accessible and transformative tools, powering everything from customer support and content generation to complex, industry-specific applications in healthcare, education, and finance.
Case Studies
In recent years, fuelled by advancements in LLMs, the numbers of AI note-takers has skyrocketed. These apps are increasingly tailored to meet the unique needs of specific user groups, such as doctors, sales teams and project managers.
