Can GPT o1-preview Build a Python Prototype of Legacy Astronomy Software in 1 Attempt?
Вставка
- Опубліковано 18 вер 2024
- Software gets old. Programming languages die out. However, there might be code out there that we would like to bring into the modern day. Can ChatGPT lead us there? In this example, I see if ChatGPT o1 can possibly build a prototype Python substitute for an old astronomical software my PhD advisor wrote in the 1990s.
GitHub Repo: github.com/kyl...
Kind of blown away you are not impressed. Not only did it run in the first go after the most basic prompt I’ve ever seen, it created a full program with a ton of features in a an incredibly short amount of time. Something that takes months to year to create was done measured in seconds. It doesn’t matter there were small bugs at all. Absolutely incredible. Plus that was not a bad gui either
Think about it this is just o1 preview ! Imagine full o1 !
To make a perfect copy in Python, take each function/method of the program and have it rewrite it in Python, that way it has manageable chunks
Great suggestion!
Yea, that's how I do it. Makes gpt hyper focused on a single problem. Nice.
Test o1 so we can come up with a unified theory 😬😬😬
That was honestly mind blowing for me. 1 prompt and you got ALL THAT?! With just a few more it could be completely working.
I make a 100.000 line page for my work using only chatgpt and Claude
Almost 7 months of works but is almost finished.
And now with the new gpto1 preview is gonna be amazing
I was already able to complete full research projects on Jupyter notebook using the Legacy/OG GPT-4 ( before it got slightly nerfed). I can only guess how powerful it has become now
I wonder if ChatGPT 4o could do this…
I really like that you're exploring the limits of the new o1 models. Keep it up!
That's the plan!
What I actually experienced with GPT-4o is the following:
You start with a prompt to write a function/class/program whatever. It does that or when failed you correct mistakes or return error messages. When done, you iterate with further requests for the function/class/program ( now it also should be abler to do this) and so on. But on a certain point it actually forgets some requirements that were already implemented and throws them out again.
One has to be really carefully when iterating on the same code over and over because there will be misses.
Well it definitely produced a better image viewer than I (software engineer) could have done ... IN 18 SECONDS!
Edit: The replies so far seem to indicate that my above comment was misunderstood. I'm saying that writing that code from a single vague prompt, in 18 seconds, is already a superhuman achievement. Even though not a finished product, it's surely is a good start and a source of ideas for a human to take on, and iterate on? As a SWE of over 30 years experience, I can say that I would have dearly loved to have access to an LLM like this earlier in my career.
I'm not SWE but maybe you just... Bad software engineer? (sorry)
I don't mean you should write whole program in 18 seconds but about a finished project by yourself
If you want your Brain Computer to go faster, just buy more braincells
1:25 literally every college student right now 💀
Haha 🤣 the go to resource
Also love the SSJ Goku pic!
Hey! new sub here! came from Wes Roth youtube channel. he did a youtube video about your PHD Code test on Chatgpt o1.
It seems like a lot of the functionality is already present in astropy and pyqtgraph, the libraries it used. But this is still impressive.
Put the code in a document and have it read the document. Generally I found it was easier to "prep" it by saying "I will upload a document for you to look at" and just have it read it. From there give it your goal, and personally I would say something like "Given the code I shared, are you able to write a program with the same functionality in python?" And then Iterate from there. But if you have working code that may be overcomplicating the process.
Thanks for the tips! I’m still so new at prompt generation, so I’m taking all the suggestions to heart!
Currently, the o models don’t accept documents. Future versions will.
@uwepleban3784 Ah, then I suppose going back to 4o would be needed. Or, it may be ugly, you can tell it that you will share code in multiple prompts and not respond until you shared it all. I had to do that a few times as well.
@KMKPhysics3 If direct document sharing is not possible, just let it know you will be sharing a large amount of code over multiple prompts. It may respond, but you can basically ignore the responses until you finish sending the code. Essentially, as long as it is following along properly, you're ok. You can tell when it's not where it needs to be.
is our curiosity and enthusiasm with testing it like this simply improving it? you're not blown away right now because it's not doing large complex tasks easily, but all of these "minuscule" additions and improvements add up over time which will do something mind blowing. i feel like we adapt to small changes over time
Don’t try to 1 shot it! You have to go back and forth with it until you get what you want.
I agree. it's amazing that we can achieve this with just a single, ambiguous prompt, especially considering this technology didn't exist before September 2022, and now, ironically, everyone is already a critic despite its groundbreaking nature.
Oh I know! I just wanted to see what would come out after 1 shot, I didn’t really have the time even with ChatGPT to recreate Aaron’s code from scratch
I have some idea for future video.
Do you have any ideas of real life astronomical problems, where
we can build up on ChatGPT coding capabilities?
ChatGPT can help easily provide height quality gui app + logic, in what areas can we apply it?
Hmm, most of my ideas would be through data analysis, but maybe we could consider using synthetic data of astronomical images per say?
Sure, it would really cool to see! To buld something usefull and professional using AI
Nuts :)
It's more impressive that it can retrieve what it was trained on easily than it is to say it wrote the software.
Incredible man, keep up the great work! I love to see the journey you're going through stress testing o1 for research applications
You should compare it with Sonnet 3.5, it’s really good at Python
If I can get my hands on it, sure!
@@KMKPhysics3 yes please try, it's just 20 per month as gpt or you can use it for free but gives less prompts
@@KMKPhysics3 Maybe you can try to email Anthrophic and ask for free API access for this kind of content ;))
My workflow has evolved with a detailed description of a project to get a high level scaffolding in which o1 will give a series of detailed tasks with prompts that then get piped into sonnet 3.5. o1 is close to being a competent ai agent as it's very close to being able to have full closed loop automation which is what I assume openai is envisioning when they say "ai agents". Like that workflow behind the scenes is ~1 year away to being able to envision, code, test, and deploy a non-trivial app completely automatically. The rate of progress is honestly faster than I had anticipated.
o1 is impressive.
I'm reading that the current best workflow is initial draft by o1-preview or mini; then fixing errors with Claude 3.5 Sonnet. It seems to work wonders.
Not impressive?! It coded the entire app in 18 seconds and he could have easily iterated on it from there and got it to improve / fix / add features and fix bugs. It's insanely impressive.
This workflow makes sen
Kyle... A tip... After you generate app code, ask the LLM to generate appropriate Mermaid Diagram markdown text...
🙂
Oh cool! Never knew that, but will do!
@@KMKPhysics3 great tip. And... also have the LLM explain the functionality it implemented - even a short user guide. I have it generate all the READMEs for any app I create using AI and it works like a champ!
Nice Death Star Mic!
Haha I never thought of the Blue Snowball like that but now I’ll never unsee it
You might want to look into what Python libraries in uses into running the AI codes. Keep in mind that there are a lot of Python libraries that is openly and easily available that would do a lot of the grunt work calculations and manipulations that your advisor might have to do from scratch back in the days. This is one of the reason why so many people across all disciplines are using Python nowadays.
In the following videos, could you continue to ask him questions from different areas of physics?
As a problem on special relativity, one on general relativity
So we still remain in the astrophysical field but we We can continue to put this new model to the test
Sure! That’s a great suggestion, and I’d love to try it.
You have been mentioned in a video by Wes R
oth and in tweets
Oh cool, I’ll have to check it out!
If you feed ChatGPT with all the mathematics/physics up until the 19th century, would it come up with the theory of relativity?
No xd but programming isnt that hard LOL
@@prohibited1125 very unrelated answer, just to boast yourself LOL.
@@prohibited1125 Programming isn't hard. Designing a system that meets requirements and then implementing the software so that it keeps to those requirements and is structured so that it can be easily maintained for an extended period - now that's hard!
I think context size would be a major impediment (probably not the only one either).
@@dante0817 take it easy baby
lol
i love your backdrop
Thanks!
You could try to upload the code as a textfile with 4o, ask something generic and then switch to o1-preview and tell it to convert the code.
Keep it up legend 🔥 Thank you for being a spark to wake more people up to the changing AI world.
There is a video that just came out today from a former Google exec named Mo Gawdet. Watch his latest video on his Chanel.
It's the best description of what the modern future of AI looks like.
You have a voice now brother and can help show more people what's possible. Keep up the great work! 🙌🏾
Thank you so much! I hope to keep informing others of what these model’s capabilities are!
Thank you uncle astronomer, very interesting!
Would be cool if you compared its response to Claude 3.5 Sonnet, just have a quick look at that option's output as it is the best coding chatbot at the moment, better then o1 I would say, unless of course the task heavily involves reasoning.
By doing a slight deviation we can better understand where the reasoning component is very beneficial in these tests and when it isn't as beneficial.
What do the ROI and Menu buttons do? Also ChatGPT is trained to first only give you sort of like a starter thingy.. but if you want it to output something much more sophisticated, you need to do more prompting.. the best in your case would be to copy his code and just say to make the python code of the same complexity and functions and make sure that the code is about the same length or longer (this will force it to produce a longer code - it still be much shorter, but yeah..). And you can then just spam "not enough, upgrade more please" and it will produce better and better code iteratively.
This one isn't really impressive, as it's basically just a simple library wrapper. Even GPT-4 could do something like that (possibly even GPT-3). This means that none of the controls that manipulate FITS image were created by it
Interesting, so the data for this model must be old; PyQt is into v6 now and the output it gave was on v5
You did not try the menu options or the ROI.. 😢
Can you please solve aptitude problems using this. Please. Good level aptitude questions
Can we progamer like Adobe. Without having to pay for Adobe 😂
❤
This is not very impressive tbh, compared to the other things you made it do on the other videos
Fair point! Just trying to demonstrate its potential in other areas than just problem solving! I think with growing performance of these models comes growing expectations
@@KMKPhysics3 People should be happy when AI isn't impressive, impressive Ai means your employers should just strike a deal with ai for the high paying management roles rather than humans
U don’t seem to understand the concepts off LLM. If there is already a tool program or whatever it is most def in the training data. So what you are doing here is using collective human generated data and selling it just like the OpenAI pr team. There is no interlegence at all and never will in the concept of llm. IT JUST PREDICTS the next token after token. Also the so called reasionhg “feature” is no more than token intensive and repeating it self output bc it doesn’t know what’s the llm will write……
Seems like an ad for OpenAI.
lol what a crybaby
Если бы он рекламил chatGPT, то не использовал бы GEMENI и Claude в своих видео
На его самом популярном стриме где он написал код, который сам писал год, он много раз упомянул Claude и использовал gemeni.
How do you know that intelligence is not just estimating what the next action should be? Human intelligence could very well just be our brains firing neurons in a specific way to “predict the next token” with tokens being “real world” actions, such as producing a specific activation pattern(idea/thought) or making some body part do some specific movement. You cannot say that this has underlying intelligence or not because no one has defined the process of intelligence, we only define it based on outcomes/behaviors. It displays intelligent behavior, so according to our current understanding of intelligence, it is intelligent. Even if it isn’t “actually” intelligent, that doesn’t matter as long as it displays intelligent behavior because that’s where the vast majority of its utility comes from.
Also, if it turns out that human intelligence is fundamentally different than LLM intelligence, that does not mean that LLMs are not intelligent. There could very well be multiple ways of achieving intelligence. Wood fires are fundamentally different from LEDs but both can produce light. Similarly, LLMs may function fundamentally differently from human brains but both could produce intelligence. Limiting our definition of intelligence to processes within brains is reductive.
Even if you're right, it's still transformative technology. What about emergent intelligence... isn't that the great discovery here?
@@foshizzlemanizzle4753 You don't know what you're talking about. Read some good philosophy of mind, like Evan Thompson and Robert Rosen. Intelligence requires autonomy, self-organization, auto-hetero-affection, inner purposiveness, in short: the system would have to be alive in order to be actually intelligent and not just an example of what Rosen calls "psychomimesis". If I have to create a prompt in order for it to generate an output then it's not intelligent. In fact, a plant is more intelligent than AI, since a plant has more autonomy and self-determination than AI. Will AI ever become actually intelligent? Maybe, but I doubt it. If so, it would have to be alive. Read som good philosophy of biology instead of claiming that we don't know what intelligence is.
Why not use GPT to make a new astrophysics discovery? Perhaps if we asked gpt, it might be up for it... pick one and get GPT to come up with a hypothesis... perhaps reanalyze existing theories...?
Dark Matter
Dark Energy
Black Hole Information Paradox
Quantum Gravity
Primordial Gravitational Waves
Baryogenesis (Matter-Antimatter Asymmetry)
Exotic Compact Objects (e.g., Quark Stars, Gravastars)
Cosmic Inflation
Multiverse and Higher Dimensions
Magnetars and Extreme Magnetic Fields
Why not talk about Cryptonica again? I thought you were in the know.
You need sleep, Kyle. Rest your weary head.
I have been very tired as of late 😢
nothing can be fit into a normal practical work flow . it's just a toy .
Cryptonica has already achieved great results, but you don't talk about it. Why is that?
I'm not impressed it basically translated part of the program into another language because that's what llms excel at.
As a software engineer, I am impressed. The work needed to write this code by hand and, without GPT, would take significantly more than 1 hour to code.
Like, but I think I'll stick with Cryptonica; I don't need anything else.
Lol, bro, better record a video about Cryptonica ATM, that's real money out there