How to Use Your Voice as Input in Unity - Microphone and Audio Loudness Detection
Вставка
- Опубліковано 15 лют 2022
- In this video I'll show you how to check the loudness of your microphone and of any audioclip. We will use this system in two examples, Making a character talk and detecting when your player blow !
▶ SUBSCRIBE : / @valemtutorials
▶ PATREON : / valemvr
▶ DISCORD: / discord
▶ MAIN CHANNEL : / valemvr
Download the awesome finger snap audio used in this project :
drive.google.com/file/d/1ZlXR...
#vrdev #valem #madewithunity
This is brilliant, a very straightforward way of integrating microphones as a mechanic in a VR game. Thanks so much!
This video killed 2 birds with 1 stone for me.
1: Is IS possible to just GET microphone input in unity without relying on some external library.
2: You can just MAKE audio clips from float arrays, and overwrite them as they loop in real time to get a continual stream of audio. (Which makes me suddenly understand one of the old VRchat voice bugs a LOT better)
FINALLY I can comfortably start tinkering with real-time audio processing. Voice modification or even complete re-synthesis. Detecting phonemes, pitch, tone, all by evaluating that float array from the microphone samples. Then generating a new float array based on samples, filters and processing rules, packing those into a new audio sample and playing that instead of the direct mic input.
Naturally there's a lot of other little details to work out, like using a Fourier transform to find the fundamental frequency of the voice sample for pitch detection, and basing all the other evaluations off that pitch so it's consistent no matter who's talking.
This is mostly a pipe dream project.
I saw a similar video some time back and it took time to understand. Now I have just seen yours and it´s perfect! I´ll try to use it for a proyect that has been in my mind time around. Thanks!
Works amazingly! Thank you. I wrote it to affect the emissive texture as well so that I can make things flash when talking or playing audio.
This came at the perfect time for my next project!
we need the finger slap in Spotify LOL. THANK U SO MUCH your videos are literally the best and they're helping me with my graduation project !! THANK YOU!
very easy to understand and straightforward. Thank you, internet stranger!
Thanks for watching internet stranger !
this is so cool!! thx mr. genius
NOICE! Thanks for this!
Thanks Valem
I saw the title, clicked, heard the voice, and thanked god Valem was the one to make this tutorial. You're going to be HUGE someday man! Like VR Brackey's
well explained!!! thx
Thanks. I'll try it.
It's an unbelievable turial, thank you very much, it help me a lot in unity.
Dang, I want to be as cool as you. This tutorial will really help me with one of my CompSci dissertation projects-- which is basically making a sound based platformer game.
Is it possible to do something similar for pitch?
Thanks Valem, This is really helpful.But I don‘t know why should use" clipPosition - sampleWindow" to calculate a startPosition?
This is brilliant! could you also share how you made the mouth movement so smooth? It looks like they all want to know how the 'smoothness' variable is used in your script. 12:10 (lower right on the screen)
I'd really like to know this too please.
tell me, did you manage to deal with smoothness?
@@MetalMonstrumGear Yes but with my own method which I'm about to share. I made the y scale of the mouth gets smaller little by little on every update. People, let me know if following method worked for you too. add this code in Update function. --> if (loudness < threshold)
{
loudness = 0;
//to prevent jittering issue, slow down the speed when the mouth go back to default size.
if (transform.localScale.y > 0.2f)
{
transform.localScale = new Vector3(transform.localScale.x, transform.localScale.y - 0.08f, transform.localScale.z);
}
}
fantastic! any chance you can give a hint on calculating the smoothness for the mouth animation?
You can set a "goal" float for the mouth size, and then approach that goal over time using delta time instead of applying the loudness directly to the mouth size.
Hi Valem, I got it working, but only if I disable the first cube that has an AudioSource as input. Do you know why could this be happening?
Finally got around to setting this up and in typical fashion I ran into an error that was my own fault.
Turns out if you try to make an audio clip with a length of 0 (Because you obsessively expose every value as a variable in the editor, then forget to actually SET said values) problems happen. xD
This happens all the time for me too glad you fixed it
cool!
Hey, i have a small issue with sockets and Photon multiplayer. When i put an object in a socket, the synchronisation with photon works perfectly, the issue is that when i take back the object from the socket, the object stays in the socket for the other players and the photon synchronisation stopped working.
Thank you so much for the good tutorial. If Android allows microphone access, is this also applicable in Android environments?
It works very well on Windows!
Thanks Valem, i got several questions.
1. I don't understand how startposition = clipPosition - sampleWindow works. If you start from position 4 and remove 64 samplewindow aren't you in a negative position?
2. Why do you return an totalloudeness / samplewindow instead of just returning totalloudness?
3. Source.timesamples Does it give the specific time per frame of the sound? (If so it can miss several sample not?
2.
Hi man thanks for the commment I'll try to answer these questions below.
1) Yes you can have a negative start position, to counter this issue you can return a loudness of 0 in that particular case. Thats what I did at 4:07
2) totalloudness / samplewindow gives a mean value around the data we collected. A mean value will give us an 'idea' of the loudness of the audio around the clip position. A mean value is computed by taking the sum of all element in the array and divide by the number of element in the array.
That's why I divide by samplewindow which is in this case the number of element we have
3) I'm not sure to understand this question correctly but Source.timesamples gives the timesample the audiosource is currently at in the audioclip that it is playing. So using it in the update function I can get the loudness of the clip at the exact moment the audio is at.
How do i know what mic sensitivity should be if everyone has different loudness
Which editor u r using and can i use it today also
hello valem i am a new game developer students so basically I have an idea to make 2d horror game sound detection where if player make sound through their mic the circle detection will get bigger and enemy can detect it.....so my question is can this tuto be used in 2d game or is it only specific on VR games....i saw that on the last part you make the mouth moving but im not quite sure.
Can You elaborate more on the particle system? Great video
Thank for tutorial , then does it works on Android device ?
How would you get the average loudness of the clip?
to make the mouth open without jitters, i used the vector 3 move towards function.
EXAMPLE
transform.lossyScale = Vector3.MoveTowards(transform.lossyScale, loudneess.position, scaleSpeed);
prolly not exactly right but i had a different use case
hope i helped!
I know im late but is there any way to translate this for 2D?
hi, is that support for unity webGL?
For me my microphone works when testing connected to pc but when i build to quest 2 my microphone is not sending sound (and ive given the app permissions)
Hi. Thanks for the video. @ 4:30 seconds into the video, you create a component called 'ScaleFromAudioClip' and have this code: 'public AudioLoudnessDetection detector;'. However, my c#
says that the namespace cannot be found. What do I need to get this error fixed? Thanks!
Are you using visual studios? If so, go to the error and it should give you the option to auto-fix the problem. In this case, it would add the appropriate namespace at the top of your script. I hope this helps. Last but not least, google the error number and you should get the correct namespace to type at the top of the script
I'm having the same error unfortunately. :(
I'm having the same error unfortunately. :(
This means your IDE isn't detecting the class you should have created at the start of the video. Check that you've created the AudioLoudnessDetection class with no typos, and that it's saved to a file called AudioLoudnessDetection.cs in your project.
How can i get a string of what are you saing on the micro??
Are there any scientific paper about the use of Unity for collecting sound triggers, and it's reliability?
la, do you have the project? to test it, it's not vr right? I'm from Argentina, I'm not a programmer
How did you smooth it out? You never told us how to do that, but we can see the smoothness factor in your video....
It's probably using Lerp like he did with changing the cube size. The more smoothness, it just means the time to Lerp is longer.
could you use it to save a voice command something. You save your voice and then compare the words on the mic with the clip so you can fire off say something like command open main menu?
Good idea ! There is the speech recognition api from windows that you can use to recognize some specific word in unity : ua-cam.com/video/29vyEOgsW8s/v-deo.html
But it does not work an all platform. Meta has enable voice recognition in the oculus quest lately but I'm not sure we can use it ourself with the Oculus sdk I'll have to look at it. :)
@@ValemTutorials thanks let me know I am interested in that. Also have you been in NEOS vr app. in there world builder you can add code to 3d objects so players can build there own addons to things. Would have any idea how to start that. I would love to add this my app. If you think it is out of the scope of your tuts would you be interested in side work?
Does Unity have a way to save a recording to an audio file for use later?
You mean record the player microphone to a wav file ? This project works well : github.com/ah1053/UnityRecorder/releases/tag/v1.0
How can this be done with PlayOneShot audio clips, though? I want to add some simple lip sync to my avatar, but not using the microphone. Instead, I want to add voice acting.
Another question: How did you add the smoothness to the code?
does anyone know how to do this but when you talk it changes a material?
I have the error, "Microphone failed. result=25 (Unsupported file or audio format.)" any help?
I have the same error. Did you find a solution?
There is no answer anywhere for smoothness.
In discord too =(((
Outstanding! Hey, Doesn't work for webgl.
But it works perfectly building for PC Windows ".exe"
Does anyone get its work on Mac? Seem like the clip position always return 0 with Mac built-in microphone
omg this accent is super sweet, i can not concentrate
If I install the game as apk to my oculus quest 2, the microphone wont work, How should I deal with it(
why did you used sample window as 64, i have copied the code word to word but my cube in scene isn't scaling. i am using audio from stream
Same
Does this work for Android too?
I wish i had any fuckin idea how to script in C# cuz Id want to modify this to be able to drive a parameter used in the animator from 0 to 1 depending on loudness- 1 being the loudest- but I cant figure out how to do that for the life of me TwT Any help w that please?
how to make it 3d?
How to add alan voice unity lip sync
can anyone paste the code he
re
Anyone can give me the script pls?
Nice one! You use a Smoothness factor in your MOUTH example ua-cam.com/video/dzD0qP8viLw/v-deo.htmlm13s How do you calculate this factor?
I would also like to know this.
Is it same like my talking tom?
using UnityEngine;
public class AudioLoudnessDetection : MonoBehaviour
{
public int sampleWindow = 64;
public float GetLoudnessFromAudioClip(int clipPosition, AudioClip clip)
{
int startPosition = clipPosition - sampleWindow;
float[] waveData = new float[sampleWindow];
clip.GetData(waveData, startPosition);
float totalLoudness = 0f;
for (int i = 0; i < sampleWindow; i++)
{
totalLoudness += Mathf.Abs(waveData[i]);
}
return totalLoudness / sampleWindow;
}
}
public class ScaleFromAudioClip : MonoBehaviour
{
public AudioLoudnessDetection detector;
public AudioSource source;
public Vector3 minScale;
public Vector3 maxScale;
private void Update()
{
float loudness = detector.GetLoudnessFromAudioClip(source.timeSamples, source.clip);
transform.localScale = Vector3.Lerp(minScale, maxScale, loudness);
}
}
legend
what about other scripts?
For anyone who hasn't been able to get their "loudness" variable:
Microphone.GetPosition requires a string for the device name.
I'm not sure why his script worked for him but I had to change it to Microphone.GetPosition(microphoneName)
I'm gonna be the bigger dog here, since i have a dog in this fight, there is missing a code:
void Start()
{
MicrophoneToAudioClip();
}
i didnt saw it, so don't complain if it's already in the video
What's your dog name ?