Create Your "Small" Action Model with GPT-4o

Create Your "Small" Action Model with GPT-4o
👊 Become a member and get access to GitHub and Code:
/ allaboutai
🤖 Great AI Engineer Course:
scrimba.com/learn/aiengineer?...
🔥 Open GitHub Repos:
github.com/AllAboutAI-YT/easy...
📧 Join the newsletter:
www.allabtai.com/newsletter/
🌐 My website:
www.allabtai.com
I try to create my own "small" action model based on Python and the GPT-4o API. Will it work? Lets find out
00:00 Small Action Model GPT-4o Intro
01:48 GPT-4o Action Model Code
05:54 Testing the Model

[@ShpanMan]

This is actually really impressive. GPT-4o watching you act and understands what is done, then writes code to reproduce it, which can then be run and automated.
Very clever flow, OpenAI should definitely hire you.

[@MilkGlue-xg5vj]

Anyone can do better than this with a powerful language model, it's not much. It's just that the rabbit is overrated.

[@clumsy_en]

Cool experimental project and idea 👍 The entire process can be scripted further to continuously store the most recent number of screenshots in 2-second intervals to VRAM using PyTensor, and a call can be triggered at any time with keyword through mic input or keys shortcut to send it to gpt-4o to retrieve the "reply last action script" and then automatically execute it to save time doing some mundane tasks👍👍

[@georgestander2682]

Thanks, this is interesting. I was wondering about this as well and had a thought about adding log data of user interactions to give the model more telemetry. So it not just vision but also the actual logs of all the interactions happening in the background.

[@ewasteredux]

Are there any local LLM's this might work with?

[@PanduPandu-fh5tk]

Maybe, LLaVA 13b can

[@mikew2883]

This is awesome!

[@cyc00000]

So good to see you getting onboard the rabid r1. It's seriously going to change lives.Enjoyed the video man.

[@TTOnkeys]

I can think of so many uses for this. Great work.

[@nic-ori]

Useful information. Thank you!👍👍👍

[@user-yw9us2qo6g]

Looks great

[@NetHyTech]

Bro Plz create video for real time vision and response

[@lokeshart3340]

Woh woh look whos here bhai kya aap mere ko jante ho ya yaad rkhe ho?

[@BThunder30]

Interesting project as always.

[@gnosisdg8497]

so where is the code for this project! looks fun

[@ibrahimaba8966]

Very interesting. I think it could also be useful to provide it with the mouse positions between different frames.
To go further, we could create multiple actions and then implement a RAG that allows the model to choose the correct snapshot and execute it.
Thanks for this video.

[@Soft_Touch_]

I've been thinking recall and omni screenshots were ways to create large pratical data sets to train lams. Do you think that is what's happening? You seem to be doing a smaller version of this

[@carstenli]

Great start. What's the GH url for subscribers?

[@futureworldhealing]

learning how to be data scientist 80% from u bro haha

[@avi7278]

honestly more legit than scammer Jesse Lyu and RabbitR1 garbage hardware scam after his NFT game scam.

[@darthvader4899]

How does it know where to click though? Does

[@kalilinux8682]

Humane and Rabbit watching this and raising another round of funding

[@lokeshart3340]

Hello sir can u recreate gemini vision fake demo in real life

[@JNET_Reloaded]

the github is always the same repo btw itl be easyer tomake a new repo for each project and put project link in description

[@wurstelei1356]

I think you can link to git sub folders. The repo is pretty messy, but keep in mind, this is free. Thou I am also not able to find code for some projects on that repo.

[@spencerfunk6697]

So literally open interpreter…