Thanks for doing this! I don't mind the frame blending at all and I think it works great as a test before going through the process of flowframes. Great job on the videos as well im glad I found your channel. ✌️
maybe the background can be more consistent if we use inpainting, that way we can make the Stable Diffusion have high cfg scale on the person and low cfg scale on the background, keeping it more consistent between frames. Great video!
@@emrek7475 I don't think they meant batch. You would have to inpaint each frame individually. You could use ControlNet to set the frame with the best background as the reference and use canny or depth, then inpaint the background area of the frame you want to modify.
the flowframes looked warped and worse to me. it still is a useful tool to know that it exists. also if you are using a video editor like blender there is no frame interpolation blending, so this is a really great tip!
In After Effects, you can adjust the settings for frame blending by applying timewarp. Put your video in a composition, increase the speed to 200%, precompose it and put timewarp on it. For example, putting the block size up to for example 11 you don't have much of the warping displacement anymore
Excellent choice for source test.. Tiles and repetitive patterns are the hardest... I think I like the FlowFrame better .. A side by side would have been good to see.. and a diff overlay would have been cool too, to see how different they are. Thanks for posting
You definitely got the spirit, and haven't seen your newest vids maybe you polished your edits, but for this one is really frustrating that for more than half of the time, it was just "umm.. wait.. eh.. ok... uh... what.. oh... ok... wait... where did... oh... got it.... uh huh.... mmmm... ok... wait... let's go back.... now.... let's see...."
If it is of interest I have almost managed temporal cohesion with stable diffusion. I posted some of those videos recently. Will add flowframes to my process and see how it goes.
I've tried interpolating the results, but I honestly like the look of it being at half frame-rate. I think it adds to the look of transition into an animated look from real footage.
I think you need to modify your technique. Go to 1/4 or even 1/8 frame. Get rid of the most inconsistent frames. Use flow frames to interpolate every other frame or every 1/4 frame. Take you most consistent frames and use those for keys in Ebsynth
for reference, here's the kind of results that can be achieved with those varying frame counts used as ebsynth keyframes. ua-cam.com/video/dwabFB8GUww/v-deo.html . promising I think. downside is I haven't figured out a good method to automate ebsynth yet and manually through the gui it seems to have a 20 something key frame limit.
Optical Flow is kinda "smarter" in that it tries to do some math but it can result in weird movement in your footage. A lot of times Optical Flow is often better than Frame Blending in Premiere, but it depends on the content and sometimes it gets really weird. I don't use After Effects yet but yeah Optical Flow in Premiere is pretty good most of the time in my book. Frame Blending is just kind of like the basic no-nonsense interpolation, but at least it is consistent. I don't think FlowFrames was really an improvement on this video, but maybe it works better with other content. Also there is a "tweening" plugin called Twixtor for Adobe products that works great but is expensive, but it works great to slow down footage.
I feel interpolation and frame blending is where working with AI animation fails miserably... I know a lot of people like that video game super warped look. I feel when AI can solve this issue i definitely will be out of the job as an animator haha... Great comparisons though...
Dude I 100% agree. I see so many people posting stuff with that video game warped look. I've gotten really good results with Disco Diffusion & Stable Diffusion by lowering the movement speed (z translation around 1), increasing the previous frame strength pretty high (~0.75-0.8), and then using Google's FILM interpolation to smooth it out.
You refer to it as "frame blending" but you are using "optical flow" in After Effects. Frame blending pretty much creates a dissolve between the two frames which sometimes could be better if you don't want the glitchy artifacts that optical flow creates.
I was going to ask about this because I think that a lot of times Optical Flow is often better than Frame Blending in Premiere, but it depends on the content and sometimes it gets really weird. I don't use After Effects yet but yeah Optical Flow is pretty good most of the time in my book. Frame Blending is just kind of like the basic no-nonsense interpolation, but at least it is consistent. I don't think FlowFrames was really an improvement on this video, but maybe it works better with other content. Also there is a "tweening" plugin called Twixtor for Adobe products that works great but is expensive, but it works great to slow down footage.
I’m sure it works better in that circumstance. I only tested it this way because people were suggesting I try Flowframes instead Frame blending to accomplish smooth AI animations.
Thanks for doing this! I don't mind the frame blending at all and I think it works great as a test before going through the process of flowframes. Great job on the videos as well im glad I found your channel. ✌️
🙏🏽🙏🏽🙏🏽
you can try to lower the input FPS at 10FPS for exemple and then use FlowFrame to up to 60FPS.
Good stuff! Thank you
maybe the background can be more consistent if we use inpainting, that way we can make the Stable Diffusion have high cfg scale on the person and low cfg scale on the background, keeping it more consistent between frames. Great video!
Hello could you please elaborate this. How can we use inpainting while using batch img2img?
@@emrek7475 I don't think they meant batch. You would have to inpaint each frame individually. You could use ControlNet to set the frame with the best background as the reference and use canny or depth, then inpaint the background area of the frame you want to modify.
It's working thanks my friend
the flowframes looked warped and worse to me.
it still is a useful tool to know that it exists.
also if you are using a video editor like blender there is no frame interpolation blending, so this is a really great tip!
In After Effects, you can adjust the settings for frame blending by applying timewarp. Put your video in a composition, increase the speed to 200%, precompose it and put timewarp on it.
For example, putting the block size up to for example 11 you don't have much of the warping displacement anymore
It’s pretty close. Probably will just have to decide which looks best on a case-by-case basis.
Excellent choice for source test.. Tiles and repetitive patterns are the hardest... I think I like the FlowFrame better .. A side by side would have been good to see.. and a diff overlay would have been cool too, to see how different they are. Thanks for posting
You definitely got the spirit, and haven't seen your newest vids maybe you polished your edits, but for this one is really frustrating that for more than half of the time, it was just "umm.. wait.. eh.. ok... uh... what.. oh... ok... wait... where did... oh... got it.... uh huh.... mmmm... ok... wait... let's go back.... now.... let's see...."
Thanks for the feedback. It was definitely one of my earlier videos, I’ve drastically improved my presentation since then.
I've just started used Flowframes a few weeks ago. Wish I got into it sooner.
If it is of interest I have almost managed temporal cohesion with stable diffusion. I posted some of those videos recently. Will add flowframes to my process and see how it goes.
I've tried interpolating the results, but I honestly like the look of it being at half frame-rate. I think it adds to the look of transition into an animated look from real footage.
Love your videos amigo! Thanks for sharing. Just curious in this case did you use EBSynth or you created each frame in batchimg2img?
Thanks man! I just used batch img2img, no EbSynth in this case
I think you need to modify your technique. Go to 1/4 or even 1/8 frame. Get rid of the most inconsistent frames. Use flow frames to interpolate every other frame or every 1/4 frame. Take you most consistent frames and use those for keys in Ebsynth
for reference, here's the kind of results that can be achieved with those varying frame counts used as ebsynth keyframes. ua-cam.com/video/dwabFB8GUww/v-deo.html . promising I think. downside is I haven't figured out a good method to automate ebsynth yet and manually through the gui it seems to have a 20 something key frame limit.
I actually planned on doing a video of using Stable Diffusion and Ebsynth together. Maybe next vid.
Bro I actually did the ebsynth method yesterday and i have to say its way better than messing with ALL the frames.
Ebsynth is amazing for this technique. Sometimes even one frame is enough for the whole shot to produce better results than batch processing
Great video! Thanks for sharing. What is the difference between frame blending and optical flow in Premiere Pro?
Optical Flow is kinda "smarter" in that it tries to do some math but it can result in weird movement in your footage. A lot of times Optical Flow is often better than Frame Blending in Premiere, but it depends on the content and sometimes it gets really weird. I don't use After Effects yet but yeah Optical Flow in Premiere is pretty good most of the time in my book. Frame Blending is just kind of like the basic no-nonsense interpolation, but at least it is consistent.
I don't think FlowFrames was really an improvement on this video, but maybe it works better with other content. Also there is a "tweening" plugin called Twixtor for Adobe products that works great but is expensive, but it works great to slow down footage.
@@kaizen5023 Thank you very much for the reply! I will continue using Premiere Pro. I also don't think FlowFrames made such a difference
I feel interpolation and frame blending is where working with AI animation fails miserably... I know a lot of people like that video game super warped look. I feel when AI can solve this issue i definitely will be out of the job as an animator haha... Great comparisons though...
Dude I 100% agree. I see so many people posting stuff with that video game warped look.
I've gotten really good results with Disco Diffusion & Stable Diffusion by lowering the movement speed (z translation around 1), increasing the previous frame strength pretty high (~0.75-0.8), and then using Google's FILM interpolation to smooth it out.
Impaibting technics for background and use a deflicker for better results
You refer to it as "frame blending" but you are using "optical flow" in After Effects. Frame blending pretty much creates a dissolve between the two frames which sometimes could be better if you don't want the glitchy artifacts that optical flow creates.
Oh ok. So frame blending if when you click it once right, and the second click is optical flow?
@@enigmatic_e Yep! As far as I understand it. Both seem to have their pros/cons.
@@ElectricFrisson thank for the information. 🙏🏽
@@enigmatic_e Thanks for the tutorials, been following along!
I was going to ask about this because I think that a lot of times Optical Flow is often better than Frame Blending in Premiere, but it depends on the content and sometimes it gets really weird. I don't use After Effects yet but yeah Optical Flow is pretty good most of the time in my book. Frame Blending is just kind of like the basic no-nonsense interpolation, but at least it is consistent.
I don't think FlowFrames was really an improvement on this video, but maybe it works better with other content. Also there is a "tweening" plugin called Twixtor for Adobe products that works great but is expensive, but it works great to slow down footage.
I am using Topaz AI for it, 15fps input to 60fps :)
Maybe try a regular live-action video for the example on the next test?
I’m sure it works better in that circumstance. I only tested it this way because people were suggesting I try Flowframes instead Frame blending to accomplish smooth AI animations.
Hard to tell a difference. I tried all 4 or 5 models in Flowframes and couldn't really tell any difference between them either.
Hi, are this animations created from a single picture, or from a video simulating rotoscopy?
Its from a video using img2img in stable diffusion local
What about Twixtror?
it looks like original choppy video is better than flowframe warping effect for this type of video
For me it looks like Flowframes has a lower resolution/quality.
Theres also a bit of a color change. Like a bit less saturated.
you can save it at RAW Avi file, so depend of the output video format.
someone tried dain too ?
Yeah. But it takes about a 100 times longer to compute