Відео

Glad you're enjoying the videos. This one isn't part of a series, but if you look through my channel you'll find AI training videos centered around robots, maze solving, and even Atari games like Breakout. Have fun!

@robertcowher sincerely amazing, and is it tutorials? Can we get a code along? Because I want to make an AI for a project that uses fundamental analysis

Glad you enjoyed the series :)

shiet I put memory_capacity = 1000000 :))

Yes. The easy answer is to reduce the buffer size. It’s what uses most of the memory. Keep in mind the buffer is there for stability so reducing it too much can have a negative impact on training. Play with that value and see where it gets you.

Are you in the training or testing phase? If you're in training, it should stay blank. If you're in testing, check the render mode.

@@robertcowher I was in testing phase. I dont know if this is because of the code or the hardware on my laptop... In the 1st run after restart the laptop ALE can show breakout, but for 2nd time it show nothing

@@robertcowher ah, I fixed it, it is because of the env.ale.* , I have to use env.unwrapper.ale.* instead

@@NhanLe-lq7xs Nice!

So what you're describing is a career journey, not a single project, and I'm not there yet but I can describe the path. If you wanted to do something like this in the real world, your path would be 1) Learn ROS, to control real world robots. 2) Learn how to physically build a robot arm. 3) Learn how to simulate your robot arm in ROS and Gazebo. 4) Learn how to port your simulated arm into a Gym-like environment for reinforcement learning. 5) Update this approach to work with your new environment and train the robot in simulation. 6) Take that trained model and move to your physical robot to complete training. If you're ready to start working with real robots, Antonio Brandi's Udemy courses are a great place to start - www.udemy.com/course/robotics-and-ros-2-learn-by-doing-manipulators

@@robertcowher Thanks a lot sir. It means a lot 🙏🙏🙏🙏😊😊

I haven't played with VLM's before but it sounds like an interesting concept. Is there a specific environment or challenge you'd like to see solved?

In your g++ command, make sure "-lcrypt" is set. Missing that could cause the error you're describing. I'm guessing you're either using VSCode without my tasks.json, or using another IDE.

So for now, I'm not even worrying about task execution order, just whether or not it can take an environment with multiple goals and accomplish them at all, and letting the environment hand out one reward per successfully completed task. I ended up taking a long break from the problem, but I'm going to pick it up in the next week or so and take another swing at it. Sometimes it helps to just..not work on something for a little while and look at it with fresh eyes.

Good catch! Fixed.

Glad to help ;) I had a lot of fun with this one.

I'm not building one specific to robots, but your comment sparked an idea and I've been working on a custom Pygame environment. At some point in the next month I'm hoping to put out a full series on building the game from scratch, converting it into a gym environment, and then training an agent on it.

Great questions. 1) For hinge cabinet, what behavior are you seeing? Have you tried deleting the weights file for that network and retraining? 2) You can use whatever IDE you're most comfortable with for most of my videos. I switched over to VSCode full time because I needed something that supported Python, Jupyter notebooks, C++, and ROS and integrated well with ChatGPT. Jetbrains(the company that makes Pycharm) has solutions for all of those problems but it pushes you into their paid tier and they bundle some of those capabilities into different products(PyCharm v.s. CLion). 3) That's something I'm currently trying to figure out. When I do, I'll definitely make a video :)

I'm very excited to receive your reply, thank you very much, and I look forward to your new work@@robertcowher

Glad you're enjoying the videos. I've also had a hard time finding good robotics-specific RL content. I may not be the best person to fill that gap, but I'm definitely having the most fun with it :)

Good observation, and the answer is simple enough. The environment returns different observation shapes for some of the goals. You can absolutely take the last state, instead of last action, and generate a new action from the new policy, but it will be the wrong shape for some of the models and cause the agent to crash. I've been playing with solving that problem other ways(for example, padding the observations so all goals/models have the same shape), but that didn't make this implementation.

@@robertcowher Thank you, it is now clear to me. You talked about your omni agent in the last video; for this agent, you might have a large observation space consisting of the observation spaces of all objects in the environment.

Great question. I'm not collecting visual data for this one, but the observations include a numerical representation of the environment. What you'll find a bit later in the course is that I'm using a hybrid approach, starting live SAC training with a buffer of experiences, and weighting the buffer towards picking up more of those live experiences for the first few hundred episodes.

I haven't tried MAMBA, but I'll check it out, and make a video if it looks interesting. Thanks for the tip!

Hey, thanks for the problem report. I wouldn't have noticed for weeks. :)

Correct. Though in this case, the super policy is going to be static(i.e. looping through a list of goals and selecting the appropriate sub-policy). We'll get there in the next few videos :)

Great question, and yes, but there's some nuance. Every timestep is considered a memory so continuous joystick actions tend to generate a lot of them very quickly. My average was right around 200 timesteps per task once I got good at it, so filling a 30K memory buffer takes about 150 successful completions of that task. I experimented with human memory buffers from 20K to 60K for various tasks and found 30K to be a good minimum buffer size to succeed at all tasks. To that end, the can_sample method we've coded here looks for batch_size(64) * 500, or 32,000. You could tweak that multiplier and experiment with less.

I was going to do this at the end of the series, but I went ahead and pinned a comment with my data set. I still recommend doing some of your own data generation to get the full process, but it's there to save you some time.

Glad you enjoyed the series :) Any specific environments you'd like to see solved? I'm already working on object sorting with a virtual Emika Panda robot arm, but that's been a challenge so far.

Hey, me too :) Honestly though, UA-cam helps me feel like there's a clear end goal when building these projects, and that's a big part of why I'm doing it. If it's useful to a few people, I've hit my goals.

If you enjoyed these, you might like the new series I'm working on. It's a similar robotic arm in a much more complex environment with a human experience component. ua-cam.com/video/ma6fbvy77Uo/v-deo.html

Glad to hear it, and great question. What you're describing is an observation wrapper. They're part of the gym library and here for exactly these kinds of changes. An observation wrapper let's you "wrap" the environment, and then add your custom logic on top of it. My recommendation would be to get a wrapper working for something really simple(say, doubling the reward when the agent hits a ball) to make sure you've got the wiring right, and then getting into the logic of what you really want to do. They sound complicated, but a basic observation wrapper can be 6 lines of code. Here's the doc to follow -> gymnasium.farama.org/api/wrappers/observation_wrappers/

Also worth noting - An observation wrapper is what I should have used instead of processing the observation in the training loop.

That error usually means that the output of one layer doesn't match the input of the next layer. Check for typos in the layer input/output sizes, and if nothing jumps out at you, start printing out the variables and their sizes as you go through the forward method in the network. One of them is likely going to be a different size than expected and that's your culprit.

Absolutely. Glad you enjoyed it.

Would you mind posting your code to GitHub so I can do a quick comparison?

@@robertcowher I have posted my GitHub link 3 times now and keeps getting deleted. Can I give it to you in any other way?

That usually means you've either typod a dimension, a batch size, or missed one of the torch.unsqueeze lines. General advice - 1) Go through it line by line and make sure you didn't miss anything. 2) Print out what's being passed through the model. Look at the actual output and see if it makes the error make more sense. 3) Plug the whole block of code + the error into ChatGPT. It's really good at spotting these kinds of issues.

Glad you're enjoying them, and I've noticed the same trying to learn this stuff myself. There's plenty of robotics content(ROS2, etc) and plenty of information on RL, but it's hard to find information on the intersection of those two topics. I'm working on getting SAC working with more complex tasks(like object sorting), as well as using human experience(piloting the robot by hand) to bootstrap more quickly. I'll post that project if I can get it running.

Out of curiosity, how much training time did you give it, and did you put your code in a GitHub repo somewhere? I don't mind taking a quick look at it. If you'd rather share that information outside of the UA-cam comment section, I just set up a Discord server for discussion on these projects, and AI/autonomous agents more generally. You're welcome to join there for assistance as well. discord.gg/dnhsk3pD2V

It wolud be even cooler if i read the video description before asking the questions. Python 3.9.16 gym - 0.21.0 tensorflow: 2.10.0 ale-py: 0.7.5 keras-rl2: 1.0.5

@@Ankara_pharao No problem at all(and sorry for the late reply). In newer videos, I'm including versions, and my requirements.txt, in the video, because you're not the only one to follow up and ask. There's also a discord now for Q&A. discord.gg/dnhsk3pD2V, and if you haven't watched anything of mine in a while, I just finished posting a series on Maze-solving with SAC, and another on robotic arms with TD3.

So, the string of hyperparameters is mostly for my own convenience. I tend to run lots of experiments, and they all take 10+ hours, so I forget what params I ran them with if I don't put together some kind of output. I'm able to reproduce something similar to what you're describing by removing the "global_step=i" variable from writer.add_scalar. That's the part that actually tells it which "step" it's on, and lets it show data over time. That line should end up looking something like this - writer.add_scalar(f"Score - {episode_identifier} alpha={alpha} - beta={beta} - batch_size={batch_size} - Critic AdamW - l1={layer1_size} l2={layer2_size} noise={starting_noise}", score, global_step=i). Something else you'll notice, I always go directly to the "Scalars" tab, and set smoothing to 0.95 or so. The "Time Series" tab has a bunch of other options I don't try to mess with.

Apologies for missing this back when you posted it but yes, most of my code is running on 3.10 or 3.11, and I haven't tested for compatibility any later. I should probably start including compatible python versions for new projects.

My GitHub is a bit disorganized, but yes. Please note that this was the initial implementation I based the videos on, so it should be very close but may not quite be line for line. github.com/bobcowher/duelling-dqn-breakout-pytorch