The most interactive CV ever made (Metahuman + AI + Microsoft Azure)
Вставка
- Опубліковано 10 січ 2023
- I had much fun preparing this and constantly switching between technologies and contexts. From conceptual and UI work (Figma) to 3d modeling (Blender, Substance Painter, etc.) to reading books about human anatomy and animating characters (especially facial expressions) to weeks spent in Jupyter Notebooks and Python (working with NLP and then performing AI experiments and training different Machine Learning models) to ONNX to Unreal 5, MetaHuman and Blueprints to NVidia Omniverse to Front-end-development in (P)React and Typescript to Azure Function Apps (partially serverless) with NodeJS to real-time video communication with WebRTC to custom signaling server to Microsoft Azure where everything is hosted to DaVinci Resolve, Fairlight and Fusion (to prepare this promo movie).
- Фільми й анімація
Can you PLEASE make a course about how you did that? I would be so happy to purchase that immediately as you really are an amazing teacher - which I am really comfortable to say after watching your FSpy tutorial.
Great job. Wonder at what point the robotic recruiter will be ready to talk to your metaverse digital twin. It will be interesting to see how that chat pans out. :)
;-)
Would absolutely love an overview of how this all works. I'm tying to build an Unreal based chatbotk and you've nailed this.
I described the MetaHuman part here forums.unrealengine.com/t/metahuman-tips-tricks/717924
Still can't find time to prepare tutorials though ;-)
this is next level !!!
holy shit this is unreal, great job
few questions:
- do you think something like Synthesia/Movio type avatar of yourself would better in this case?
- are you aware about deepfaketom? do you think with all these tools you could get close to similar results?
These tools are great if all you need is a video. In this case I needed a functional application with full control over my Avatar, interaction and smooth "transitions" between the answers.
hello there, this is amazing, and I could see you've put a lot of effort in building this CV. I have few questions and some work, if possible can we connect somewhere?
Nice work !!!
Thank you ;)
Very very!! Good job!
Thank you Rodrigo :)
Wonderful, curently, i am in journey build like this, could you give me some clue or recourses how to build this?
This is so interesting. May I ask how did you get access to the Audio2Face SDK? Did you apply thrue NVidia program? Would like to test it for business applications.
Thank you Rando. I have applied to different NVidia programs, but as far as I remember Audio2Face and other Omniverse tools are publicly available
I'd love to see this in action. Can I get limited time access?
Do you ever plan to open source this project? Would love to contribute
I keep wondering that this man might be working with Soburo Arasaka🤣
can u Create a github repo or a YT tutorial on how to create this??
If I find enough time I’ll make a series of tutorials
Are you going to sell this your 3D model? How can I contact to you?
To be honest, it's just my CV. I haven't been thinking about turning it into a product.
If you wanna get the metahuman looking less like a 3d model, and more like a actually face; try putting a live deepfake, over the metahuman, of your face! - Should really help.
I decided not to because I needed a real-time app, not a video.
@Bartosz Pampuch You can do it all in real-time, deepfake and all, unless I'm misunderstanding you; and since it's your face, you can take and use HD images for it too.
I'm talking about a live deepfake, in which it puts the deepfake on you, or in this case, your 3d model, in real-time!
@@pinkdonkey4397 Do you have any links to a convincing real-time deepfake? I mean there are a lot of issues (e.g., visemes must precede phonemes), which means you have to delay audio a little bit. Not a big deal, because usually we're talking about a few frames, but still. Another problem is the compute power that is required. I pay for the GPU cloud and the GPUs are responsible for handling 3d app and WebRTC encoding, I'm afraid running a deep-fake AI on top of it would effectively mean less simultaneous streams a single machine can handle. I'll have to verify how much GPU it consumes (not for this project, but for the future). Thanks for sharing your ideas
@Bartosz Pampuch Yeah, it might not be idle for your situation, proformence wise.
I'll have to do a bit more research, I'll get back to you soon!
Thx
can I have your email?
Sure, but I'd rather not share it here, under the video. Send me a DM on twitter.