Introduction to Deep Learning Models in Unreal
Вставка
- Опубліковано 25 сер 2024
- In this session we will explore different ways deep learning neural network models can and will be used in gaming. We will dive into different use cases and show how to leverage existing open-source model zoos. Then we will demo how to apply a PyTorch Style Transfer Model that does real time inferencing frame by frame. We will learn about the new experimental plugins, Neural Network Inference (NNI) that is powered by ONNX Runtime and OpenCV Helper that were released with Unreal Engine 5.
Join the Mixed Reality Developer Program to get the latest on our developer tools, events, and early access offers:
aka.ms/IWantMR
/ onnxruntime
onnxruntime.ai...
/ onnxruntime
/ 77691267
github.com/mic...
Participate in the Mixed Reality Dev Days Online Hackathon now through July 8, 2022. Register to join at aka.ms/DevDays...
![The moment we stopped understanding AI [AlexNet]](/img/n.gif)
The door has just opened and it is very interesting to see how far creativity will bring AI to gaming in next few years.
Cool stuff! Keep pushing the envelope. 😊
Thanks for this great info. Keep up the good work!
I’d start using it just as I’ve said with the Cesium plug-in getting deep learning to look at satellite imagery to create photorealistic worlds on the fly so you can zoom in anywhere using Google maps / the cesium plug in
And instantly have a photo realistic world that you can visit in the oculus rift
Just by recognizing the colour values and attributing them to the correct landscape layers creating masks on the fly at runtime
Then using foliage on the layers automatically placed
And based on WHERE you are on the map the program will know what kind of foliage to spawn
And attribute to the colours and landscape layers
Thanks for the talk, it's certainly very promising tech!!
Can anyone provide more info on the "Automated reachability testing" shown at 11:37 ?
I'm interested in using reinforcement learning inside Unreal Engine 5.
Correct me if I'm wrong, but I think you can't train anything with ONNX runtime and you need something like PyTorch to train.
Trying the project on 5.2, it looks like the NNI is hooked up, but everything looks brown.
Is there any super-resolution demo in unreal plugin?
If people can do RL in any game enigne, the npc's behavior is absolutely going onto another level. This can be super big. If that really happens, npc can really give out information even the grammar can be broken for some reason. For now, the only way to give information in games is to manually script them.
I personally don't care about the graphics. Minecraft is good enough. No matter how good the games look, what happens in them is always so boring.
I'm just busy trying to adapt communication with NPCs through machine learning so that they can meaningfully communicate and carry on a conversation within the lore of one world.
What to do if i want to run deep reinforcement learning in unreal that learns on the fly?
nice
Please provide support for MicrosoftOpenXR plugin for Unreal
got so far just to not figure out the style node doesnt connect to neural net work variable
U figure it out?
Once you add megascans assets and the entire library free to the unreal community
“Text to unreal level API “
would be amazing
Adding the Cesium plug-in
You could literally say
Make the city of Toronto as an old Western theme, add a small population horses buggies
Done.
It’s almost that easy manually
Especially once you add audio effects and visualizer affects and convolutional neural network abilities
You could give the computer a line of spoken text
And not only will it create the scene in photo realistic detail
You can then use those images those renderings as source images for even more deep learning imagery effects
And adjust it and control it with audio
(I’ve been watching “the egg 4K” lately, so now I’m trying to re-create it I’ve actually re-created that video in unreal
So now my desire is my own way of adding my own source images to an audio program like WZRD, Using source images rendered by the unreal engine. That’s how I found you today)
Or you could provide one source image either a rough sketch or an actual photograph of a scene
And the AI will recognize it and convert it into an actual 3-D scene
By using shape recognition and object recognition and pulling in the correct assets and then modifying the assets to match the photograph Upon rendering
So the AI is modifying the assets even things like hair so that once rendered it matches the photograph pixel by pixel
If done correctly you could get deep learning to watch a movie
Then automatically re-create the actors and locations and animations and lip-synchs and re-create the entire thing as a 3-D world and experience
I’d like to see deep learning take the cesium plugin and satellite imagery and shape / object recognition and automatically populate unreal levels with mega scans assets so trees are where trees should be and bushes are where bushes should be
In photo realistic detail because it’s the unreal engine and megascans assets
You would essentially rip the colour data /texture data into colour based masks so it’s grass Road rock Beach
Automatically create “brushify” layers in the landscape material and then naturally use megascans surfaces as the image textures
So now you’re not looking at satellite imagery, you’re looking at unreal landscape layers with tiling and bump mapping and all of it in photo realistic detail automatically
So now you can give the unreal engine Vincent van Gogh starry night as a source image
And all of your renders will have that style
And you can create the scene itself using text to unreal levelAPI
Cool
I’m gonna go cut my ear off right now
Then you tell it to add a character
And referencing the meta-humans
The AI automatically creates anyone you mention basically
Anyone you can supply a photograph off, a reference of, it will take that image, all the images you can find on the Internet and train it to facial recognition them and map them to that Metahuman face
Convolutionally adjusting the mesh, so the face and the render matches the reference photograph
Now you can almost create anywhere on the earth in photo realistic detail with photo realistic people
Using a few lines of text