World’s Fastest Talking AI: Deepgram + Groq
Вставка
- Опубліковано 11 бер 2024
- - Deepgram: tinyurl.com/deepgram-aura to get $200 free credit
- Code Tutorial + Overview: github.com/gkamradt/QuickAgent
OVERVIEW:
I’m Greg Kamradt, and I’m on a mission to figure out how businesses will create more value using AI. In this overview, we look at the 3 pieces needed to create a super fast AI voice bot.
Sponsors that help support the channel:
- Deepgram (Transcription Services): tinyurl.com/deepgram-aura
- Deepgram's live demo: aura-tts-demo.deepgram.com/
Groq®:
The Groq® LPU™ Inference Engine is the fastest language processing accelerator on the market, delivering 10x faster inference.
groq.com/
GREG’S INFO:
- Twitter: tiny.one/sIY2j61
- Newsletter: tiny.one/vXzrYJ3
- Website: tiny.one/T948oRT
- LinkedIn: tiny.one/knMMWIw
- Work with me: tiny.one/6AZ890O
- Contact Me: Twitter DM, LinkedIn Message, or contact@dataindependent.com - Наука та технологія
Deepgram is the most slept on AI player.
yeah they have a ton of good data to work with
Thank you for creating this tutorial it's exactly what I was looking for. Great content!
Thanks Greg you do alot for the community ,I have respect for you did in semantic chunking in langchain repo
Nice!
Looks cool. Thank you for showing this idea. Will definitely implement it in my project.
Nice! Good luck! What is the project you're building?
@@DataIndependent I didn't started yet. But I will start it soon.
I built a voice bot with this exact set up steaming and all quite a few months ago and the biggest issue was the latency! So excited to see this is no longer a Problem!
I experimented with integrating Groq with Siri, and it was quite interesting. The response time was pretty impressive.
Great heads up on new STT/LLM/TTS technology. I had been working on a Unreal Meta Human demo which got pretty close to RT using a Google STT/ChatGPT/TTS. One of the other things to think about if one want's to get into 2D/3D talking head chat apps is streaming back viseme data as well as TTS audio. Plus maybe emotion tokens of some kind. I can't wait for all of this to be offered on one platform/api service.
this is exactly i am looking for .
thank you youtube.
love you lots
Nice!
Great video, Greg!
This is exactly what I wanted to do this weekend. Great timing.
Same, I want to try to do translation (language identification per speaker / switching whisper model could be tricky)
nice to both of you!
@@DataIndependent I ended up pulling down your project and ran into a few issues, fixed them and made a few improvements like response interruption. It's pretty good but groq being limites to llama 2 and Mistral is a shame. It is at least able to fill the time of latency of the more powerful models so that at least it still feels more natural even if the first 10 seconds are filler until openai or Claude 3 ingests input and starts streaming.
@@avi7278 That's awesome! If you were open to doing a PR to add interruptions I would definitely want to share it with the community.
Did you add filler too?
@@DataIndependent it's on my list for this weekend, cheers, appreciate the starter.
I purchased a ReSpeaker Mic Array v2.0 for this purpose. It captures speech with great clarity. Works out of the box on Linux, so should be able to make a standalone voice assistant with it.
Cool that sounds good, please share when it's out
Awesome, subscribed on the strength of this video.
Ha thank you!
Great stuff! Did you not cut in the video to reduce the waiting time for the LLM response? If you did not, the speed is impressive! Thank you so much!
Super fast example - really highlights the power of this tech
Big time Cam!
Thank you! Amazing!
Great video, Greg!
You gave me some interesting ideas :)
Awesome - excited to see what you build
Wow - incredible!
Nice!
Awesome Greg! Best TTS-STT demo yet. Do you have any ideas on how to modify your example for two people having a conversation, and the AI participating as a third person. For example, debate students are debating and want the AI to be the judge to help them improve their debate skills. I would love to hear your thoughts on this. Thanks for this tutorial. I've been looking for this solution since the 90's!
Thanks for this! Testing it now ... the TTS audio isn't streaming and im struggling to fix it but ... THANK You for sharing your code! Super helpful and informative video!
I’m in the same situation, got the listening and response working great but no TTS. Spent all day breaking and fixing it 😂 still confused why it won’t talk! Great guide (I needed python 3.11 to get I back working and had some issues with dotenv, ended up hard coding the APIs) - it’s a great piece of work! 👍👍
Try these updates
github.com/gkamradt/QuickAgent/commit/21ae2b0e286759e186e12a76addd250a5a491381
Holy hell that is incredibly fast
Right? Ya it was super cool
conversational endpointing is a great idea, but I'd like to see that combined with a small model agent that was only looking for breaks in the conversation and an appropriate time to interject. Maybe with a crude scale for the length of the response. So if the user has a break in the point they're trying to make - we don't want the user interrupted and the conversation moved on - what would be more appropriate would be a simple acknowledgement. But once the point is complete, we would then pass back that we want a longer response.
This. I need something like this for a project, but I'm not very familiar with Groq or Deepgram yet; just starting to dig in. This thing starts responding with the first little pause, so it constantly cuts me off when I'm just pausing momentarily to think of how I want to phrase the rest of my sentence. If it wants to send data at every minor pause in order to understand context, predict the full query, and begin formulating a response, that's fine-- but it needs to wait until I've finished my entire input before verifying/sending its response. Out of the box, this is like a person who doesn't actually listen to what you're saying and is just waiting for their turn to speak. Is there an easy way to affect the response times and/or understanding of when the user has finished a full thought or do I need to develop logic/rules from scratch?
@@frothyphilosophy7000 not that I’ve seen. And this would be a massive leap in improving conversation. It literally just needs a small model to parse the text at every pause and see if it’s an appropriate time to interject. Just the same as a polite human would do. The groq api should be able to do it. I’m really surprised we haven’t seen this effectively enabled anywhere yet.
@@HideousSlots Gotcha. Yeah, guess I'll need to implement that, as it's unusable otherwise.
Rasa NLU would be good for intention detection like greeting, closure , and the domain or custom intentions
I'm not familiar with that - thanks for sharing I'll check it out
This is great, thanks for the awesome demo. I’m not a tech person, but can I clone my sales agent’s voice using ElevenLabs and integrate it to this process? I guess I can train the LLM to respond in his tone and sales style? Thanks
Yeah you could for sure. It would take a bit of work back and forth but it’s there
Thanks! How does it compare with assemblyai universal 1? do you know which speech-to-text support arabic with the best accuracy in real time?
Amazing video.. Just wish it had a walk though of instructions
ElevenLabs turbo with deepgram stt. As tested. Cant beat it.
FYI you dont need to use an API for TTS or Speech to Text both can be run locally using Faster Whisper for TTS and Coqui for Speech even if you dont have the worlds most expensive GPU because both of those only use a couple of GB of Video Ram. Going forward, on device will be the way to go for TTS and STT because they simply do not require that much processing power.
Nice - what kind of latency are you getting with those? Along with accuracy
There is also Piper for tts, it's open source and wirks realy well in my non-gpu laptop
there is a caveat with ur solution becuase fatser whisper does not detect silence out of the box and hence, you wouldnt know if the user has finished talking or not, which Deepgram does.
@DataIndependent great video! Do you know how to get ChatGrok / Mixtral working with Langchain 'bind_tools'? I'd love to swap out my ChatOpenAI for ChatGrok if possible!
I haven't tried that out yet - sorry!
I miss your videos!
More coming!
very nice, many thanks. latency of the llm still way too high. maybe 6-12 months to bring it down to
This is exactly what I was looking for. Thanks. But the text to speech functions seems to be not working in my windows. I tried writing the audio in a wav file (for debugging purpose), but the file is also can not be played. I thought may be codec issue, so I tried to convert the file in online to mp3, but got an error message showing " Invalid data found when processing input".
Any idea, how can I make the text to speech function working? Another note: it does not show any error message in terminal, it just does not play the audio.
Same issue here. Hoping for an answer, too.
Try again with these changes
github.com/gkamradt/QuickAgent/commit/21ae2b0e286759e186e12a76addd250a5a491381
Wow that was fast… Grok literally went open source 😮
I was able to sneak early access for the video
do you know if any of the open source models (STT and TTS) with Groq give the same latency?
My guess is no - oss usually isn't as fast as paid
Hi Greg, Rutvik here. We have created something similar to this but using GPT as LLM and Elevenlabs as TTS. We are facing issues with Silence detection with Deepgram. I know you did mentioned in your video at 3:53 that we need to make sure that we dont talk too slowly. And unfortunately Deepgram only has MAX value of 500 ms for endpointing (Silence detection). Can you confirm if we are using proper configurations with Deepgram? Following are my configurations.
'punctuate': True,
'interim_results': True,
'language': 'en-US',
'channels': 1,
'sample_rate': 16000,
'model': 'nova-2-conversationalai',
'endpointing': 500
Could you help me understand what's going on. I'm running this in Docker and keep getting an error when it gets to running the .py: "ModuleNotFoundError: No module named 'langchain_groq'"
on windows, having some problems with libraries. After fixing all dependencies, etc. I think I'm having problems with ffplay for the computer to send the audio to the speakers. Taking a look. Somehow I can't listen to ffplay.
Debug the response from TTS without streaming first. I had to change the URL (it was some beta URL), they have changed it now and the one in Github right now returns an internal error (non 200 response), so the audio won't stream.
Both good points, I'll update the URL at least
@@DataIndependent I tried it on Ubuntu Linux and (I think I) also have a problem with ffplay. However, somehow while it does convert my voice to text and LLM responds with text it does not convert text to sound. To test it I did different checks to verify that the connection with Deepgram is working. I used a test script where I used the Deepgram API with a text message and then I streamed the audio directly back to ffplay for playback and it worked. So that is why I am puzzled why the code is not working on the Linux side. Any help is appreciated. I truly enjoy your thoughtful tutorials and videos.
There was an issue with payload. When I commented out the "voice": self.MODEL_NAME it worked. I also changed to pygame from ffplay on Linux. But that may not have been the issue.
I suppose Whisper/ tiny model would be faster than this Deepgram ? have you tried?
Nope not yet - let me know how it goes
Can this be added to a website?
how could we interrupt the voicebot can anyone help (pls)
Thank you very much for the vide, how can I integrate RAG into this pipeline?
Put it in the LLM step, set up a retriever and then call to it
@@DataIndependent Yes, thank you very much.
vectordb = FAISS.from_documents(all_splits, embeddings)
retriever = vectordb.as_retriever()
BUT LLM doesn't accept retriver param,
So I used ConversationalRetrievalChain
Am i the only one not able to hear the TTS back even if i installed the ffmpeg? What am i mising? i tried both from vscode and from normal terminal.. nothing plays out, no errors though. connections to deepgram is checked and successful.
Any hints?
I’d also love to know this. I’m in the same boat! It works .. just silently for me too
I updated the code, try again or make these changes yourself
github.com/gkamradt/QuickAgent/commit/21ae2b0e286759e186e12a76addd250a5a491381
Getting error when firing off the "request" in "speak()" - "err_msg":"Failed to deserialize JSON payload. Please specify exactly one of `text` or `url` in the JSON body." It works though, if I take the "voice" attribute out of the the "payload" AND also change the model to the model in their docs, which is "aura-helios-en". Other than that, thank you so much for sharing! It's hard work!
Thanks for this - I found this same problem. I had it working in beta (before the video game out) so these were the changes needed for their prod version.
Updated!
@@DataIndependent Thanks Greg! Awesome content! I appreciate you!
how can we interrupt the ai??? plsss helpp
This app won't find my API keys either Groq or Openai though they are there. Too bad. Any suggestions greg?
API's being found either!
The original Groq demo had an impressive speech demo. How did that work? Interviewer interrupted Groq repeatedly.
Can you link it?
ua-cam.com/video/pRUddK6sxDg/v-deo.htmlsi=Kg15nRUEEr1AHTGx
How’s this compare against VAPI ?
Haven't tried that yet
Vapi is extremely expensive. I think in a couple of months we should be close to Vapis performance with open sourced models and tooling.
I have tthis error C:\Users\vnt>pip install deepgram
ERROR: Could not find a version that satisfies the requirement deepgram (from versions: none)
ERROR: No matching distribution found for deepgram
You trying to use their python-sdk? Maybe try "pip install deepgram-sdk"
Yeah - the suggestion here is the key
github.com/deepgram/deepgram-python-sdk?tab=readme-ov-file#installation
Showcase it within a streamlet app doing Langchain shenanigans! Please!
Like what kind of shenanigans?
@@DataIndependent
Starter difficulty
Steamlit app using Langchain and Deepgramdeepgram that allows you to upload a pdf / embed then when ready we can simply chat with it live , asking a question , retrieving information
Advanced difficulty / Aspirational
An agent using function calling and a set of tools like web search / calculators etc ...
if we can have an asynchronous continues conversation with one supervisor agent who asynchronously can "ask" other agents to do time consuming tasks, like being able to talk to the supervisor agent in Langgraph schema
thank you so much for this video, i have a problem when i run the scipt : i have this error message :" Could not open socket: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997) "
any help please
I had this problem a while back too and it was super annoying. The solution was a lot of googling and I think I might have even used it as a push to upgrade my python
@@DataIndependent This doesn't look good for me then, hhhhhh, was the solution obvious or did it require a lot of manipulation?
Facing WARNING: API key is missing
Could not open socket: server rejected WebSocket connection: HTTP 401
on Stt model
Ya…you’ll need an API key which I can call out better in the read me
I have my own api key with 300$ credit. Still it's showing this error
Can you please send me a screenshot of your .env(hide your api)file@@DataIndependent
I have my api key with 300$ credit but still it's showing this error massage. So can you please send me your screenshot of.env( hide your API)@@DataIndependent
Is there any alternative for Spanish? I cannot find it.
how to add those filler words?
Is there any developper who can help me to set up this?
Would be great for training lawyers to take depositions and do other oral tasks.
Yep it would… But unfortunately lawyers are probably done for… like 99% of them.
It'll help augment roles ha, not replace
i cloned the code but response time is like 12 seconds. 4 weeks past and i'm late to the party
Doesn't this suck? I've been subscribed to Deepgram for quite a while now. You'd think they would have some good competition. Alas, none that have stuck around, and open-source is infested of dragons.
as a total beginner, I can't find a way to run this.
Same here..
I'm not a coder, but can someone please build one of these that I can pay for so I can basically have my own Jarvis to talk to on my computer?
Have you tried this one?
www.hume.ai/
I can't drop the LLM below 3000ms :( maybe im fucking it up?
Nice. But in reality your demo is 3,500-4,000 ms from when you stop speaking to getting a response. It does not match the numbers you are printing...
No good can come of this.....lol
yeah - it'll be tricky to navigate but there will be use somewhere
Sir can you provide full tutorial of this plzzzz