Unreal Engine 5 - Ultimate Voice AI Tutorial - Masterclass from scratch

Поділитися
Вставка
  • Опубліковано 30 жов 2024

КОМЕНТАРІ • 128

  • @miliv_eth
    @miliv_eth Рік тому +15

    kind reminder for any security phreak who is bumping her head against the wall after an hour of debugging: please enable your mic access in win under Privacy Settings > Microphone
    : D. I almost fell down the chair when I figured. Many thanks for the tutorial btw, its truly amazing and that you published it open source. People like you are the reason the unreal community flourishes (*^ ‿ ⁢*)♡

    • @daniellopez9582
      @daniellopez9582 4 місяці тому

      I highly recommend a mic that allows to be muted from the device itself.... I don¿t trust that microsoft respects any privacy settings

  • @devon9374
    @devon9374 3 місяці тому +1

    Words cannot DESCRIBE how good this tutorial is, You are the man!
    Along with Georgi Gerganov and Georgy Treshche :)
    GOD BLESS YOU ALL!!!!!

  • @charons.5549
    @charons.5549 Місяць тому +2

    its freaking amazing, the only tutorial which is working for me!!!

  • @diadorchak9767
    @diadorchak9767 10 місяців тому +9

    For people wondering about the {Blank Audio} error I found mine was capturing the audio from this youtube tutorial. Go to the Start capture function that is copied an pasted at 8:28 in the video and set the Device Id from 0 to maybe 1 or 2 depending on your set up. For me 0 was my audio from my PC and 1 was my microphone. I hope that clears some confusion.

    • @benblaumentalism6245
      @benblaumentalism6245 5 місяців тому +1

      Not here. I'm getting [Blank Audio] every time, no matter how I set up the Device ID. I know my mic signal is getting into Unreal because I made an input level meter and used a channel from the Audio Capture component. I thought maybe that was interfering somehow, so I set up a blank project just to test it. Still nothing. I'm using UE 5.3.2, in case anyone happens to have any insights into what's going on. The Marketplace says the plug-in works up to 5.4.

    • @badashphilosophy9533
      @badashphilosophy9533 4 місяці тому

      Having the same issue. Did you manage to solve it?

  • @shaheenal-asadi
    @shaheenal-asadi Рік тому +4

    Very nice tutorial, thank you for creating this. The future of dynamic NPC interactivity is here today :)

  • @darkalden
    @darkalden 3 місяці тому +3

    Hello !!! Nice work you've done here, it's the same thing I'm working on my own although I still think the most important part is the lipsync part to give realism to the conversation. It's still very green what I'm trying to do with the lipsync (using an old OVRLipSync plugin but it can be improved and many people use it) but nothing very well finished yet. I'm working on it but if I manage to get a version out soon I'll share it with you so you can have something more complete. Good work, best regards.

  • @Smartmoufdeity
    @Smartmoufdeity 10 місяців тому +3

    Genius! Pure genius!!

  • @diluciousOK
    @diluciousOK Рік тому +3

    Thanks for creating this amazing tutorial! Subscribed, and I'm looking forward to see more of your creations :)

  • @englishwithphil42
    @englishwithphil42 7 місяців тому

    This is just magic, thank you, time to finally install UE5 since there's no plugin for UE4 :)

  • @anthonymannwexford
    @anthonymannwexford 11 місяців тому

    Excellent. Thank you. I've been looking for something like this for a while. Well done.

  • @sahurows
    @sahurows Рік тому +1

    Thank you for this tutorial, it's some amazing knowledge you are sharing.

  • @FluffyMrB
    @FluffyMrB Рік тому +1

    Superb tutorial!

  • @harithsami843
    @harithsami843 10 місяців тому +1

    Great tutorial thanks heaps

  • @ganzukc2676
    @ganzukc2676 5 місяців тому

    Thanks for the tutorial!

  • @algorithman2129
    @algorithman2129 Рік тому +2

    Nice nice :)
    Now we just need a AI like dolly locally and quite a bit smaller, fed with only game specific content. Hmmm, my todo-list is growing.

  • @张伟-l6p
    @张伟-l6p 6 місяців тому

    Great tutorial, I want to understand how you set up the digital character to change from a casual animation to an intro animation

  • @SK-hj1xh
    @SK-hj1xh 11 місяців тому +2

    Hi. Can I use my RVC voice model in this pipeline? so that my character can speak with special voice

  • @jonaltschuler2024
    @jonaltschuler2024 26 днів тому

    What an absolutely incredible tutorial! Just what I needed! Now on to lip sync with Metahumans..have you tried this? Wondering if I need to use the Text to Speech Streaming with Timestamps?

    • @MarvelMaster
      @MarvelMaster  25 днів тому +1

      did not find a fast solution for lipsync so i scrapped that

    • @jonaltschuler2024
      @jonaltschuler2024 25 днів тому

      @@MarvelMaster A lot of plugins, but not sure what’s best. If anyone has any recommendations, please chime in!

  • @SuperNikpro
    @SuperNikpro 11 місяців тому +1

    very good tutorial.
    ijust have one problem my engine crashes when i hit # multiple times
    it crashes with an access violation at 0xfff...
    any ideas how to fix this?

  • @aurelianobuendia24
    @aurelianobuendia24 9 місяців тому +1

    Amazing work, how can i Connect animations to it?

  • @istealpixel7371
    @istealpixel7371 Рік тому

    Genau das hab ich gesucht^^

  • @diegobevilacqua4407
    @diegobevilacqua4407 Рік тому

    Thank you! finally it worked, now i m trying to add a metahuman with Quixel bridge, i can add but i don t know how to start O_o using also the lipsynch like the "text to speech" function...

    • @MarvelMaster
      @MarvelMaster  Рік тому

      jeah maybe thwre is some realtime plugin for that

  • @jaymore5207
    @jaymore5207 24 дні тому

    Can we use the same architecture for open source LLM models such as LLama, MistralAI ?

  • @aika9938
    @aika9938 11 місяців тому

    Hi, this is amazing. You are a life saver. I'm facing a little problem, I've been trying to 'Get Response Content' but that node does not show, instead i'm seeing all others - 'Get Response Value' , 'Get Response Content as String', etc. What can I do and what could be wrong? I am using UE4.27

  • @maria0_0
    @maria0_0 3 місяці тому

    runtime audio and speech plugins are no longer free - is there any alternatives?

  • @1RNik
    @1RNik 6 місяців тому

    Thanks for the tutorial! For your patreon what tier would we join to gain access to the project files?

    • @MarvelMaster
      @MarvelMaster  6 місяців тому +1

      as minimus supporter 😁

    • @1RNik
      @1RNik 6 місяців тому

      @@MarvelMaster Thank you!

  • @vreality69
    @vreality69 24 дні тому

    Hello, we work on similar project and i would know if we can get meeting to discuss about that for my company.
    Thank you

  • @marijnfuhler7246
    @marijnfuhler7246 Рік тому

    Lovely tutorial, thank you for creating this. Just one question, because you make use of the VaRest plugin do you need a internet connection in order to use chat-gtp? And if so, how is the ai going to work on a build when the api is dependent on wifi connection?

    • @MarvelMaster
      @MarvelMaster  Рік тому

      yes gpt requests are made online... as well as text to speech... I think computation would be too intense locally at the moment.

  • @장유선-z1d
    @장유선-z1d Рік тому

    Thank you for the content!! It helped me a lot! Is it available on unreal engine 5.2??

    • @MarvelMaster
      @MarvelMaster  Рік тому +1

      i think. the plugins are not available jet fir 5.2

  • @joellegalloni6689
    @joellegalloni6689 Рік тому

    Thanks for this! I am not so clear about what the streaming option is doing though. Isn't this supposed to output a stream that we can play through before the request is completed? Like we start reading the stream and continue updating the playback until the stream is over? The way it is now I still have to wait till the audio is completely through before playing so I don't see the streaming part of this?

    • @MarvelMaster
      @MarvelMaster  Рік тому +1

      yes the way it is now is just wait till end... but in theory with streaming you can split inputs and outputs to make it faster... if one knows. how to do that

  • @rotv6492
    @rotv6492 4 місяці тому

    Hello does anyone know how to fix the "the audio data could not be processed to the recognizer since the thread is stopped" problem with this plugin?

  • @wizzard_design
    @wizzard_design Рік тому +1

    Is it possible to make microphone work without clicking any button but when you start game? Thanks!

    • @MarvelMaster
      @MarvelMaster  Рік тому

      you mean voice activated? Not sure if the plugin has a option for that or another plugin does that

    • @badashphilosophy9533
      @badashphilosophy9533 4 місяці тому

      Did you find your answer, id like to do that too but a bit like how we say hey google to activate google assistant, but ive got a god game so u have to pray dear god lol.

  • @AICineVerseStudios
    @AICineVerseStudios Рік тому

    great solid tutorial. seems like they have update their github page and how to use the plugin page is not there anymore. We have to create all the nodes by ourselves for the blueprint

    • @MarvelMaster
      @MarvelMaster  Рік тому +1

      no the copyable codes is in the runtime speech recognizer doc not audio importer doc

    • @AICineVerseStudios
      @AICineVerseStudios Рік тому

      @@MarvelMaster Sorry my bad! you are right. Followed the eleven labs integration. All is going well but from the eleven labs whatever is returned, its not playing from the buffer. I am not using chatgpt response but actually just passing the text generated based on voice recognition to the custom event of playing voice reply. Any help here would be really appreciated.

    • @MarvelMaster
      @MarvelMaster  Рік тому +1

      @@AICineVerseStudios you can debug print the resonse from elevenlabs... if its cryptic the itsa sound 8f its text that says some error then something is wrong

    • @AICineVerseStudios
      @AICineVerseStudios Рік тому

      @@MarvelMaster It's working! Eureka! 😀😀😀😀

    • @AICineVerseStudios
      @AICineVerseStudios Рік тому

      @@MarvelMaster Ok actually I'm building this to run on a cellphone as a person avatar. Is it recommended to follow your approach to building the same for Iphone or Android ? Can there be processing issues ? Or would it even be able to use the plugins for voice recognition ?

  • @Ysys-king
    @Ysys-king Рік тому

    Hi, thanks for the tutorial! I have a question. I created my voice in ElevenLabs but can not find voice ID. Where can I get it? Thanks for help.

    • @MarvelMaster
      @MarvelMaster  Рік тому

      the elevenlabs website surely can tell that

  • @Hunty.
    @Hunty. 5 місяців тому

    any way to do this with a local LLM instead so that it doesn't have all the filters that chatgpt does?

  • @tuargames5357
    @tuargames5357 8 місяців тому

    Is there anyway to setup a memory for the character like give him a backstory?

  • @Ysys-king
    @Ysys-king Рік тому

    Hi! Everything works fine in project. However, when I packaged project, only "VoiceRecognitionStarted" and "VoiceRecognitionStopped" are writing. So it does not work when I packaged. Do you have any suggestion?

    • @MarvelMaster
      @MarvelMaster  Рік тому

      iirc plugins need to be packed manually in project settings

  • @wolfgangschwarz9415
    @wolfgangschwarz9415 Рік тому

    Thanks so much for the effort and clearly structured tutorial! Eventually I made it work for me :) I guess you are right and this is still basic with lots of things that can be tweaked.
    My biggest wish: How can I find an ID for generated voices to implement in UE?
    Btw, the German language model works quite well and I do not see any latency difference to English.

    • @MarvelMaster
      @MarvelMaster  Рік тому

      different voice ids are in the description ir you create your own one if you habe an elevenlabs subscrption

  • @taimur879
    @taimur879 8 місяців тому

    Hi . can This be Used to generate Lip sync for non metahuman Characters

  • @estaling3802
    @estaling3802 Рік тому

    Hello, I followed the tutorial, and it only allows me to speak once. No error or string printed out after the second time voice recognition. Do you know why it's happening?

  • @user-ug6kk5ux5q
    @user-ug6kk5ux5q Рік тому

    I really love what you do, in my app I need to download the audio file and storage it at a specific file path. Do you know how could I do that ? Any small advice would be really helpful!! It is not clear to me what the runtime importer does with that file.. I need to turn it into an asset, because I have another API that requires the audio, but also the asset of the audio. Thank you!

    • @MarvelMaster
      @MarvelMaster  Рік тому

      did you check the documentation of the audio importer... there is even a setup for non streaming

  • @jang4765
    @jang4765 Рік тому

    awsome

  • @user-ug6kk5ux5q
    @user-ug6kk5ux5q Рік тому

    Hi, great tutorial! In my case the imported soundwave didn't work, so in order to find out why, I added a "Switch on ETranscodingStatus" attached to the "OnResult" event that continues from "Status".
    Well.. this switch returns "Failed to read". Any idea why would this happen? I did correctly putting the switch there? (like, should I trust it? If yes, why would it fail to read the audio file? I did exactly what you did!). Thank you!

    • @MarvelMaster
      @MarvelMaster  Рік тому

      maybe there is no audio file... did you printdebug the response?

  • @dave0602
    @dave0602 Рік тому

    Does it work for only 5.1? or include above?

    • @MarvelMaster
      @MarvelMaster  Рік тому

      havent tried other depends on if the plugins are available for newer version

  • @badashphilosophy9533
    @badashphilosophy9533 4 місяці тому

    When i press # in the blurprint search, keyboard elements isnt appearing as an option.
    Today is my first day in unreal engine so am asking here for everyone who has the same issue in future doesnt have to leave the video to debug. Thanx whoever provides the answer to this issue :)

    • @badashphilosophy9533
      @badashphilosophy9533 4 місяці тому +1

      Found it, though it may be specific to the latest version of UE (5.4.1)
      You have to type "keyboard event" and then the key you want ie keyboard event T in order to get the precise key in the search.
      If you just type the word keyboard and then #, youl have to scroll up and locate it manually, or alternatively if you just type the word key and then #, youl have to scroll down and find it manually

  • @JesseThaBest
    @JesseThaBest Рік тому

    Hey!
    So I've been looking into this Speech Recognition, is there any way to make it more efficient or faster?
    I'm noticing it takes minimum of 3 seconds for the Model to recognize what I say and return an answer, is there a way to make it "instant"? I already use the tiny model size and put english only, so can't reduce the sample size. :D

    • @MarvelMaster
      @MarvelMaster  Рік тому +1

      hm maybe better hardware? And maybe there is a way to make it recognize to get streaming answer from chat gpt and elevenlabs... meaning not waiting till the whole sentences are processed but getting the responses while they are generated partly?

    • @georgydev
      @georgydev 10 місяців тому

      Hey, I just wanted to mention that since then, the plugin has undergone significant improvements, and its speed is now much faster compared to half a year ago

  • @nBodyResearch
    @nBodyResearch Рік тому

    Hey, in your waterline video it was for ue4 and needed world displacement and tesselation. Is there anyway I could show me how to do the waterline without those options in ue5? Thanks

    • @MarvelMaster
      @MarvelMaster  Рік тому

      I read that you can use virtual heightfield mesh insteafmd of tessellation in ue5

    • @nBodyResearch
      @nBodyResearch Рік тому

      @@MarvelMaster I figured it out!! Though now I just need to figure out how to get swimming and buoyancy to work in a custom water mesh!

  • @titan3rd474
    @titan3rd474 6 місяців тому

    Great content, How to do animations when voice is playing

    • @MarvelMaster
      @MarvelMaster  6 місяців тому +1

      you can get when audio is playing and the start an animation

    • @titan3rd474
      @titan3rd474 5 місяців тому

      How can I mix the idle animation and the talking animation, is there more tutorials coming up, because this has been a great help 😀

    • @MarvelMaster
      @MarvelMaster  5 місяців тому

      @@titan3rd474 Animation is a topic itself...you can use an animation BP or just Trigger animations in the character BP

  • @vaibhavdevadiga9807
    @vaibhavdevadiga9807 Рік тому

    where should I connect Make SpeechRecognitionParameters, in order make it work? 14:40

    • @MarvelMaster
      @MarvelMaster  Рік тому +1

      there is probably a set parameters node or something

    • @vaibhavdevadiga9807
      @vaibhavdevadiga9807 Рік тому

      @@MarvelMaster can you let me know me know how to do it?

  • @ai_and_chill
    @ai_and_chill Рік тому

    i'm having a lot of trouble with creating save game slots of the chat array history. would you be able to help with that please? this video is great thanks again.

    • @MarvelMaster
      @MarvelMaster  Рік тому

      isnt it just a string to save?

    • @ai_and_chill
      @ai_and_chill Рік тому

      @@MarvelMaster i have it connected to twitch and twitch messages trigger this whole process to happen instead of a microphone. i'd like it so that when I load it only loads based on who just sent the message. so it would pull the array save game slot associated with that username. what's driving me mad is that I figured this out earlier this week and forgot to back it up. running in circles trying to remember what I did. lol

    • @ai_and_chill
      @ai_and_chill Рік тому

      @@MarvelMaster oh right i didn't explain the problem. it's saving all messages onto one array. even though I set the username variable as the save game slot name.

    • @ai_and_chill
      @ai_and_chill Рік тому

      @@MarvelMaster chatgpt and all them have been completely useless in getting this to work XD

    • @MarvelMaster
      @MarvelMaster  Рік тому

      @@ai_and_chill you can try use something different than array

  • @gscsdjoe7152
    @gscsdjoe7152 Рік тому

    i have one question, when i change language in setting, it wont work, if i change it back, the issue will be fixed, why, anyway, tks your video, its help me a lot. if u can rep me about how to change language, its will be great. and it will help a lot people just like u and me who are not the native english speaker.

    • @MarvelMaster
      @MarvelMaster  Рік тому

      depends on what does not work... you can set voice recognition language in plugin and the also use the multilingual model of elevenlabs

  • @rofethegeek5820
    @rofethegeek5820 Рік тому

    how did you manage to animate her when she talks?

    • @MarvelMaster
      @MarvelMaster  Рік тому

      just start a random animation when audio plays

  • @wolfmarkstudio
    @wolfmarkstudio Рік тому

    So cool man can you plz tell me how to mzke the animation while talking

    • @MarvelMaster
      @MarvelMaster  Рік тому +1

      just start animation when sound starts

    • @wolfmarkstudio
      @wolfmarkstudio Рік тому

      @@MarvelMaster ohh i was thinking your using npc meta plugin or sdk meta plugin thank you

  • @eunachaanakya
    @eunachaanakya Рік тому

    hello can you help for solving about "Audio_Blank"?

  • @toras99
    @toras99 4 місяці тому

    What should I do if when I start recording sound, I get the following message: you

    • @MarvelMaster
      @MarvelMaster  4 місяці тому

      new ue5 versions dont set mic source automatically, on the blueprint somewhere there is a node where you can set the audio source as number standart is 0 try 1 2 or e

    • @toras99
      @toras99 4 місяці тому

      @@MarvelMaster I changed numbers from 1 to 7 in start capture. Did not help

    • @MarvelMaster
      @MarvelMaster  4 місяці тому

      @@toras99 you can open the audio capture plugin level there you can see and select your audio devices folder should be somewhere hidden in plugin folder

    • @devon9374
      @devon9374 3 місяці тому

      Also, check the Process Audio Data sample rate. If it is zero, change it to 44100

  • @MilanJain-y4s
    @MilanJain-y4s Рік тому

    Great tutorial! For some reason, when it prints out what it processes from speech recognition (step 1 of the video), it always returns a series of exclamation points (!!!!!!!!). Any idea why?

    • @MarvelMaster
      @MarvelMaster  Рік тому

      step 1 is voice recognition... make sure your mic works and is not set too loud or quiet

    • @MilanJain-y4s
      @MilanJain-y4s Рік тому

      @@MarvelMaster right now I'm just using the mic on my Mac and am having issues - would this only work (well) with an external mic?

    • @MilanJain-y4s
      @MilanJain-y4s Рік тому

      @@MarvelMaster bump on this -- is this a Mac issue? Would I need an external mic?

    • @MarvelMaster
      @MarvelMaster  Рік тому

      @@MilanJain-y4s try it on another pc then

    • @georgydev
      @georgydev 10 місяців тому

      Hey, that mentioned problem occurred some time ago due to a resampling issue on the engine side caused by audio chunks being too small for proper resampling. But it has been fixed since then, and now it shouldn't cause these problems anymore

  • @bagusdai3684
    @bagusdai3684 Рік тому

    the audio wont play, this is the log "ogRuntimeAudioImporter: Warning: Imported sound wave ('CapturableSoundWave_1') data will be cleared because it is being unloaded
    LogRuntimeAudioImporter: Warning: Imported sound wave ('CapturableSoundWave_0') data will be cleared because it is being unloaded"

    • @MarvelMaster
      @MarvelMaster  Рік тому

      did you try google it or look into the documentation... andengine version?

    • @bagusdai3684
      @bagusdai3684 Рік тому

      @@MarvelMaster i use UE 5.2

  • @MemeIsAllreadyInUse
    @MemeIsAllreadyInUse 7 місяців тому

    how to change the mic input?

    • @MarvelMaster
      @MarvelMaster  6 місяців тому

      dont know maybe in projext settings or windows settings... usually it takes the standart mic input from windows afaik

  • @diegobevilacqua4407
    @diegobevilacqua4407 Рік тому

    Im at 20%, everytime i speak it print me "you", any ideas?

  • @user_375a82
    @user_375a82 Рік тому

    Good except the lips do not move when she speaks.

    • @MarvelMaster
      @MarvelMaster  Рік тому

      jeah unfortunately did not find an easy solution for that

    • @calvinwayne3017
      @calvinwayne3017 Рік тому

      @@MarvelMaster id like to see u do it though, metahuman comes with the visemes, so text to visemes then set the visemes, would be the logic, but thats way over my head

  • @maxschweder704
    @maxschweder704 9 місяців тому

    Awesome tutorial!