Dude, this is INSANE!! I'm in college studying computer science and this is REALLY impressive to me! It's a unique idea I haven't seen done before, keep it up!!
How on earth does this only have a couple hundred views??? The video quality, editing, voiceover are all amazing! Deserve way way more recognition my guy keep up the good work!
I really appreciate the kind words, it means a lot! I'm a small channel and am only just switching to doing content like this so it'll take a while to grow, but more importantly it seems lots of people are already enjoying it 😁
One possible improvement would be to calculate the forces working on the bar. I believe the downwards gravitaional force is constant as well as the pulling force. This would allow you to accurately calculate the displacement of the bar. The formula would be: dh = v_0*t + ( (a-g)*t^2 )/2 and dv = a*t. You could combine these with the fish velocity (v_f) and the position of the fish (p_f) to get the formula: t = ( v_f - v_b + sqrt[ (v_b - v_f)^2 - 2a(p_b - p_f) ] ) / (a-g) Where the bar velocity v_b can be either be observed or calculated by v_b = v_0 + a*dt. (t is the time the button needs to be pressed. If t is negative you do not press the button. a is the pulling acceleration and g is the gravitational acceleration. p_b is the position of the bar.) By adding the criteria that the bar velocity should be the same as the fish velocity on arrival, the following function is derived: t = -(sqrt[a(v_0^2*g+(a-g)(p_f-p_b)g+v_f^2)] + v_0*a) / [(a-g)a] This does only work if the fish is above the bar, but it is quite simple to derive a second formula for the other case.
Great idea, thanks for the detailed suggestions! I vaguely recognise the equations from when I learned mechanics but that was a long time ago. Would definitely make a lot of sense to work it out like that! I think it would also be useful to work out how much the bar bounces at the top/bottom (although it may just be a perfectly elastic collision)
You should try having it rapidly press c rather than holding it. The more you need the mar to move, the more rapidly you press c. This gives more control.
I just found this video after months of obsessing with stardew and this is freaking insane 🤩🤩 did I understand anything? Nah but I’m flabbergasted hahaha awesome work!
Stuff like this is why I chose a cs major. Haven’t done any python yet but I have some experience with JavaScript and HTML before college and I’m going C++ now. Videos like these always make me excited for my major
I absolutely love your approach to this problem! Using a software PID controller is a great approach! As a possible improvement to your algorithm, could you combine the bar velocity and the error velocity using a convolution function? That would "integrate" the I in PID and basically, the longer the fish is under the bar, the higher the output value would be. You could run a check per cycle to see if the output value increased or decreased. If it decreased, that means the fish is no longer under the bar, in which you could utilize a finding function. This could in theory minimize the time the bar is not under the fish and could result in faster fish acquisition. According to GeeksForGeeks, "OpenCV has the function cv2. filter2D() which can be used to apply a convolution to an image." Unfortunately, I'm not as familiar with the implementation of this theory, but I think it would eliminate misses when the bar is at its lowest length.
Me who has installed a mod to skip the fishing minigame after hundreds of hours in stardew: "Uh huh yep wow cool that's so neat" Seriously, though, this is a great video.
Machine learning would likely do a much much better job of keeping the green on the fish, speed of refresh is still going to be an issue I imagine, but worst case you get to play with implementing a simple ML solution which is all the rage these days.
Maybe if you use a different kind of image filter other than greyscale you could improve the speed. I think something like a bord detector might be useful
I ended up just modding the fishing mini-game out of the game after catching a few perfect catches, you no longer have to do the minigame if you hook that fish. I like not having mini-game in multiplayer, since without the minigame fishing doesn't use as much time in the day and you = the catch amounts of single player actually getting day pause while fishing.
you only really need a single line of pixels downwards through the minigame ui, right? just walk along that and ask whether blue, green or fish. no computer vision required at all. store the last known position and velocity for when it flashes and extrapolate. You control the bar, you have perfect information about where it is even with minimal data points.
Once you've figured out where the bar is on the screen, you should just scan for the first and last green pixels in a vertical line down the left part of the bar (or look for the blue/green border at most). Way easier and orders of magnitude faster than doing proper image recognition. There should be a color you can look for for the fish as well. The hardest part about the harder fish is probably knowing when to stop "chasing" the fish since some of them tend to hang out in the middle and it may not be possible to actually catch them reliably without knowing their movement patterns. I'd be interested to see if they can be reliably caught at all; sometimes it feels like you just get bad RNG on the movement. One question about this video - were you catching every fish in the final tests or were there still some misses, and if so, what were the percentages?
That makes way more sense than the way I was doing it! Now I've done more research into OpenCV it's kind of obvious haha I was curious about the harder fish as well, although I haven't played Stardew enough to know where to consistently find them. If the program is purely reacting to the fish movements rather than predicting then I doubt all the fish are able to be 'perfectly' caught every time as they seem to move faster than the bar can. Doesn't rule out being able to avoid missing any fish though! For the final tests I didn't show all the fish I caught/hooked. On the old character I believe I was level 8 fishing with no bobber help - hooked 22 fish (13 of which were perfect catches) and didn't miss any. On the new character I actually did 2 days but only showed the first day of the results as they were both similar, with around 14% of the fish being missed, and mostly due to treasure chests getting in the way! (Also fewer perfect catches with the new character)
*as a physics nerd* would it be possible to use an acceleration based equation instead of velocity? Since the bar changes velocity as the button is pressed, it is safe to assume acceleration is at play. We can use concepts from dynamics to refer to the velocity of the bar IN TERMS OF the velocity of fish (relative velocity) to determine time to press button (assume bar has constant acceleration and fish has relatively constant velocity)
Cool video, just want to throw in the comments that also Sikuli X streamlines this type of automation also with significantly better image recognition performance.
Hey, could you tell where you got the images you use? I was trying to improve your code on a different repo to make the program cast the rod and catch the fish when it bites. I couldn't find the needed images in game files :/
Hey! I took screenshots of the game while playing - you've got to make sure no anti-aliasing is used when the screenshot is taken for the image recognition to work. It's been a while since I did it so I can't remember exactly how, unfortunately. Let me know if you get it working :)
@@TheAbysmalKraken well, I was trying to avoid it but couldn’t find how to. Your images looked perfect which I couldn’t achieve but thanks for the response, I’ll keep trying
This is a nice experiment. There are many ways to improve on this. I think you should control for the bars position, not the hold time. You could either fully model the bars speed and hold time or you could train a control law. You could also add some small logic to adjust the setpoint position of the fish inside the bar to the position of the bar in the minigame to give the controller more time to react (keep fish at bottom of bar at bottom of minigame and top of bar at top)
As a programming exercise is pretty cool, as a tool for Stardew is... pretty pointless, sadly. Everyone is commenting like "why does this have so little views": because there's already a mod that autocatches with just one click.
Good question! I would think it's possible, however for machine learning you would have to run it MANY times for it to learn enough, which would take far too long without making some sort of simulation of the minigame (which would also have to be accurate). I considered it briefly, but as the problem is quite simple it makes more sense to go for a hardcoded solution if that makes sense! You might be able to use machine learning to optimise the a, b, and c parameters, but those could alternatively be optimised using something like Bayesian optimisation :)
What should I try next?
You could try to automate the blackjack in the casino
@@AwesomesauceMan2 Honestly I've never got to that point in Stardew before but I can have a look at it!
create auto fishing pixel world pls 😁
Dude, this is INSANE!! I'm in college studying computer science and this is REALLY impressive to me! It's a unique idea I haven't seen done before, keep it up!!
Thanks so much dude! I'm glad you liked it 😁 good luck with college!!
How on earth does this only have a couple hundred views??? The video quality, editing, voiceover are all amazing! Deserve way way more recognition my guy keep up the good work!
I really appreciate the kind words, it means a lot! I'm a small channel and am only just switching to doing content like this so it'll take a while to grow, but more importantly it seems lots of people are already enjoying it 😁
prolly cause the code is relatively simple, still a cool video though
One possible improvement would be to calculate the forces working on the bar. I believe the downwards gravitaional force is constant as well as the pulling force. This would allow you to accurately calculate the displacement of the bar. The formula would be: dh = v_0*t + ( (a-g)*t^2 )/2 and dv = a*t. You could combine these with the fish velocity (v_f) and the position of the fish (p_f) to get the formula:
t = ( v_f - v_b + sqrt[ (v_b - v_f)^2 - 2a(p_b - p_f) ] ) / (a-g)
Where the bar velocity v_b can be either be observed or calculated by v_b = v_0 + a*dt. (t is the time the button needs to be pressed. If t is negative you do not press the button. a is the pulling acceleration and g is the gravitational acceleration. p_b is the position of the bar.)
By adding the criteria that the bar velocity should be the same as the fish velocity on arrival, the following function is derived:
t = -(sqrt[a(v_0^2*g+(a-g)(p_f-p_b)g+v_f^2)] + v_0*a) / [(a-g)a]
This does only work if the fish is above the bar, but it is quite simple to derive a second formula for the other case.
Great idea, thanks for the detailed suggestions!
I vaguely recognise the equations from when I learned mechanics but that was a long time ago. Would definitely make a lot of sense to work it out like that!
I think it would also be useful to work out how much the bar bounces at the top/bottom (although it may just be a perfectly elastic collision)
Super smart stuff man! Love anything like this 👏
Thanks man!
WOW i think that this content is really underrated! Keep up the good work!
Thanks so much!
You should try having it rapidly press c rather than holding it. The more you need the mar to move, the more rapidly you press c. This gives more control.
Sound like a PID controller
After hundreds of hours of SDV I would confirm tapping the button is much better than holding it down.
I just found this video after months of obsessing with stardew and this is freaking insane 🤩🤩 did I understand anything? Nah but I’m flabbergasted hahaha awesome work!
Wow that was impressive! feel like my brain just increased in size watching this haha :D
Thanks Twidies!!
Stuff like this is why I chose a cs major. Haven’t done any python yet but I have some experience with JavaScript and HTML before college and I’m going C++ now. Videos like these always make me excited for my major
For sure, I love it! If you've done JavaScript and C++ I'm sure you'll pick up Python right away :)
damn! thats awesome!
Since i hate coding i did only understand until a certain part what you were doing but it does work :)
This is amazing! I just started learning python and it still shocks me how useful coding can be
this is no AI xD, still interesting
Why can't it be considered an AI?
@@TheAbysmalKraken Theres a difference between a program and an ai, this is a program ;-)
Great video! hopefully the next ones garner more attention
You are so smart, even tho i am in 5th grade and of course i wouldnt understand this, its still extremely impressive
Haha thank you! It's taken me a few years of practice and I've still got lots to learn.
I absolutely love your approach to this problem! Using a software PID controller is a great approach! As a possible improvement to your algorithm, could you combine the bar velocity and the error velocity using a convolution function? That would "integrate" the I in PID and basically, the longer the fish is under the bar, the higher the output value would be. You could run a check per cycle to see if the output value increased or decreased. If it decreased, that means the fish is no longer under the bar, in which you could utilize a finding function. This could in theory minimize the time the bar is not under the fish and could result in faster fish acquisition. According to GeeksForGeeks, "OpenCV has the function cv2. filter2D() which can be used to apply a convolution to an image." Unfortunately, I'm not as familiar with the implementation of this theory, but I think it would eliminate misses when the bar is at its lowest length.
Wow i love this! i love stardew valley so this would be nice if it was a mod or smth
I’d love to see this with Ai learning
i am a nuclear engineering student, and i am taking c++ this course but to be honest this idea blown my mind do your best you are unique
Wow, that sounds very impressive! I would imagine that involves some control engineering :) and thank you!!
I was shocked to see how few views this had when I scrolled to check the comments.
Good video! Made a difficult problem easy to understand.
Me who has installed a mod to skip the fishing minigame after hundreds of hours in stardew:
"Uh huh yep wow cool that's so neat"
Seriously, though, this is a great video.
Machine learning would likely do a much much better job of keeping the green on the fish, speed of refresh is still going to be an issue I imagine, but worst case you get to play with implementing a simple ML solution which is all the rage these days.
Maybe if you use a different kind of image filter other than greyscale you could improve the speed. I think something like a bord detector might be useful
can u make a video for how to input in the code to the file on Stardew Valley? i mean where can i put the your code to the file on stardew valley
Literally watching this while procrastinating on studying for my Control Systems exam...
Good luck! I did my degree in control systems so I know the pain 😂
@@TheAbysmalKraken I passed! Obviously watching your video helped me there ;)
That's amazing, congrats!!
Would there be any chance you would be willing to share the code for this? It looks awesome!
Of course, I've just added the GitHub link to the description :) and thank you!
i would say get rid of the print statmentrs just because it does have SOME effect on the program even if its small
I ended up just modding the fishing mini-game out of the game after catching a few perfect catches, you no longer have to do the minigame if you hook that fish. I like not having mini-game in multiplayer, since without the minigame fishing doesn't use as much time in the day and you = the catch amounts of single player actually getting day pause while fishing.
you only really need a single line of pixels downwards through the minigame ui, right? just walk along that and ask whether blue, green or fish. no computer vision required at all. store the last known position and velocity for when it flashes and extrapolate. You control the bar, you have perfect information about where it is even with minimal data points.
May i see the whole script?
Sorry for the incredibly slow reply, I've just added the GitHub link to the description.
Once you've figured out where the bar is on the screen, you should just scan for the first and last green pixels in a vertical line down the left part of the bar (or look for the blue/green border at most). Way easier and orders of magnitude faster than doing proper image recognition. There should be a color you can look for for the fish as well.
The hardest part about the harder fish is probably knowing when to stop "chasing" the fish since some of them tend to hang out in the middle and it may not be possible to actually catch them reliably without knowing their movement patterns. I'd be interested to see if they can be reliably caught at all; sometimes it feels like you just get bad RNG on the movement.
One question about this video - were you catching every fish in the final tests or were there still some misses, and if so, what were the percentages?
That makes way more sense than the way I was doing it! Now I've done more research into OpenCV it's kind of obvious haha
I was curious about the harder fish as well, although I haven't played Stardew enough to know where to consistently find them. If the program is purely reacting to the fish movements rather than predicting then I doubt all the fish are able to be 'perfectly' caught every time as they seem to move faster than the bar can. Doesn't rule out being able to avoid missing any fish though!
For the final tests I didn't show all the fish I caught/hooked. On the old character I believe I was level 8 fishing with no bobber help - hooked 22 fish (13 of which were perfect catches) and didn't miss any. On the new character I actually did 2 days but only showed the first day of the results as they were both similar, with around 14% of the fish being missed, and mostly due to treasure chests getting in the way! (Also fewer perfect catches with the new character)
*The Legend wants to know your location*
*as a physics nerd* would it be possible to use an acceleration based equation instead of velocity? Since the bar changes velocity as the button is pressed, it is safe to assume acceleration is at play. We can use concepts from dynamics to refer to the velocity of the bar IN TERMS OF the velocity of fish (relative velocity) to determine time to press button (assume bar has constant acceleration and fish has relatively constant velocity)
Since there is only a y component the equation simplifies nice to where you just need the velocities and can solve for time
this is so underrated
Cool video, just want to throw in the comments that also Sikuli X streamlines this type of automation also with significantly better image recognition performance.
When you get this thing completed I'm getting this so I can finally get all the achievements
Hey, could you tell where you got the images you use? I was trying to improve your code on a different repo to make the program cast the rod and catch the fish when it bites. I couldn't find the needed images in game files :/
Hey! I took screenshots of the game while playing - you've got to make sure no anti-aliasing is used when the screenshot is taken for the image recognition to work. It's been a while since I did it so I can't remember exactly how, unfortunately. Let me know if you get it working :)
@@TheAbysmalKraken well, I was trying to avoid it but couldn’t find how to. Your images looked perfect which I couldn’t achieve but thanks for the response, I’ll keep trying
Hey, any improvements so far bro?
@@mertnacar8296 nah bro it seems that I can’t escape aliasing 💀 I only need a clean picture of that exclamation mark when the fish bites to proceed
@@DoreminaAlso If you're using an Nvidia GPU, you can go into Nvidia Control Panel, and turn off the Anti-Aliasing
Did it cast for you to?? That was good
Heads up, this was in my recommended 😉
What a blessing, thanks for the great news!
Very pog
now make it catch an octopus or a lava eel
Bit late but great video :)
Also could you write a program that helps with beating the arcade games
do TAS speedruns try to catch fish?
It says that no module named numpy but I have numpy installed what do I do
What is the song
I've just added the names of all the songs used in the video to the description :) I hope that helps!
Use hook bobber X
Use various coding knowledge to engineer a fishing ai that is still worse than hook bobber and be like concernted Ape
This is a nice experiment. There are many ways to improve on this. I think you should control for the bars position, not the hold time.
You could either fully model the bars speed and hold time or you could train a control law.
You could also add some small logic to adjust the setpoint position of the fish inside the bar to the position of the bar in the minigame to give the controller more time to react (keep fish at bottom of bar at bottom of minigame and top of bar at top)
It's a great program.... but its not self learning so its not AI.
I wish I wasn’t dumb 💀
How to add this in Mobile stardew
As a programming exercise is pretty cool, as a tool for Stardew is... pretty pointless, sadly. Everyone is commenting like "why does this have so little views": because there's already a mod that autocatches with just one click.
Could you do AI learning to make it more accurate?
Good question! I would think it's possible, however for machine learning you would have to run it MANY times for it to learn enough, which would take far too long without making some sort of simulation of the minigame (which would also have to be accurate). I considered it briefly, but as the problem is quite simple it makes more sense to go for a hardcoded solution if that makes sense! You might be able to use machine learning to optimise the a, b, and c parameters, but those could alternatively be optimised using something like Bayesian optimisation :)
Just add a barbed hook and you win
Could have just made a bot via memory addys (fish pos + bar pos) wouldn't make for a good video though haha.
Anyone try this on 1.6 yet?
this is not an AI...
this is a bot not a ai
i fish better