DeepSeek Coder AI 🐍 The Best Coding Model I've Tested? (Open-Source)
Вставка
- Опубліковано 5 жов 2024
- In this video, we use a new coding rubric to test a coding-specific LLM called DeepSeek Coder. It's an incredible model that is fine-tuned for coding tasks, and today, we're going to see if it's as good as they say.
Enjoy :)
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewber...
Need AI Consulting? ✅
forwardfuture.ai/
Rent a GPU (MassedCompute) 🚀
bit.ly/matthew...
USE CODE "MatthewBerman" for 50% discount
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
Media/Sponsorship Inquiries 📈
bit.ly/44TC45V
Links:
Coding LLM Leaderboard - bit.ly/49E7jFH
DeepSeek Coder - deepseekcoder....
Use RunPod - bit.ly/3OtbnQx
How To Install LLMs on Runpod - • Run ANY LLM Using Clou...
RunPod TextGen UI Template - bit.ly/3EqiQdl - Наука та технологія
What tests should I add to the new coding-specific rubric?
Can you add some regex generation, explanation and regex fixing? Make sure that the questions are unique!
Build an e-commerce store with one item, many, items with features (size, color, material), subscription business; secure back-end.
CLI based todo app.
C# code tests, C++ code tests, see if it can create the snake game in Unity Engine or Unreal Engine. I tried this with GPT 4 and it failed, it was able to create the code and tell me step by step how to set things up Unity, but it failed in being able to fix an error in the movement logic. It understood the problem that I described but was unable to come up with a solution.
React app, Regex, some coding challenge from a page, etc.
VS Code tutorial would be great!
There's a million of those
Let’s make it a million and one. I for one don’t find it obvious how to run a local (or in my case cloud) open source coding LLM and plug that into VS code copilot. This would be much better that the existing copilot since you can train on your own code base (and it’s already amazing)
@@ryanfranz6715 use pieces
VSCode tutorial please
Yes please!
Yes, please. A tutorial for LLM in VS Code would be great. How do we deploy them, local Vs cloud and how would a basic workflow look like? Thank you 😇
Totally would love a tutorial on setting it up with VSC.. the least amount of money I have to give to microsoft the better.
100th agree :)
I love it!! Snake works perfectly! The 7b model is also impressive! Reaching also pretty close to GPT4 for a 7b and its also capable of creating Snake Locally!
Crazy, right?
Please make the tutorial, this looks like a ton of fun!
And if possible not only for VSCode but also for neovim. At least if there are neovim plugins for this open source models, please mention it. Cause I use a proprietary plugin for coding and would love to switch to the open source one.
Insane. What parameter model did you use for the test, you didn't say I think?
EDIT: If it was the 33B then I think it would be worth the hassle to test the other models or at least the 3B or 7B because that's I think what pretty consumer level hardware can run pretty easily
For formatting the !=, i think the term is ligature and different fonts can include them.
Ligatures are special characters in a font that combine two (or more) troublesome characters into one. For instance, in serifed text faces, the lowercase f often collides with the lowercase i and l. To fix this, the fi and fl are often combined into a single shape (what pros would call a glyph).
Awesome, thanks for letting me know.
Ah -- a ligature. Thanks for the info.
"troublesome characters" lool.. is that answer from AI haha....
funny af
I appreciate these overviews, and this is great to see! I teach undergrad CS and before teaching full-time my work was in creative text generation, creative agents, and using AI to support creativity (glad I defended before GPT-2 dropped). I've been hoping to find coding models we could run on our local servers for (at least a subset of) students to eventually use, but don't have the time to keep up this the wild progress without videos like these. +1 for sure on VS Code usage, anything on deployment and practical workflow integration is fantastic. Great vid as always!
Oh, and in terms of some niche use cases for different kinds of code benchmarks, beyond generation, code reviews, checking if requirements are met or violated, and checking test case coverage have all been almost fully hallucinations in the limited tests I've done, so these are always appreciated (assuming the model is meant to support instruct)
Hey Matthew, Really enjoy your videos. Just a question on the code challenge section. If you change the engineer prompt to "Please find the issues with this code and explain in detail: ". You will find the outcome to be more inline with expectations.
Awesome, thanks for sharing this.
Super cool! +1 for a tutorial on the vscode integration, is possible to use a local LLM for coding without the cut and paste in the files?
I'm sure it's possible, especially with the VS code integration.
@@matthew_berman If you could show us how to have a coding model understand an entire relatively large code base at the same time it would be the biggest leap in AI since chatGPT
I'm going to test this model with the continue vscode plugin. It should work and prevent some copy and paste. I'd love to see other options though. So +1 for a video
a local copilot would be wild if you can get that to work!
YES!!!!! VS Tutorial for the win!!!!!
Thank you for doing that multi-turn error testing, that's what all coding LLMs testing should go into!
The score-results vs GPT-3.5 and GTP4: I'm guessing all the scores are for un-quantized models? It would be interesting to see how the scores were for all different quantized versions of the open models too.
Awesome video, please make a video about using an open source llm as a copilot on visual studio code. That sounds very interesting.
The ≠ formatting is a rendering option. VS code supports it. Under the hood it's still the same text.
You mentioned Phind and I wanna make sure you know that while Phind did release an open model Phind/Phind-CodeLlama-34B-v2 but later iterations have all been proprietary and behind a commercial cloud offering. So when people say "phind is the best" they generally aren't talking about Phind/Phind-CodeLlama-34B-v2 rather they are taking about the proprietary commercial offering.
That's really good to know, thanks for sharing. Is there a significant difference between v2 and the subsequent closed-source models?
I asked Phind who it is... and I was told that there is GPT 3.5 under the hood, modified / optimized by the company for coding
@@matthew_berman it's known to be significantly better, but we don't know how they built it
So what's next on the game test front? Breakout? Space Invaders?
🤔
Yes please to the VS code open source code pilot. Folks working behind a corporate firewall would be rescued by that!!! 🙏
While it is impressive, it might not mean much. LLM UA-camrs have been using the same tests to evaluate models for the past six months. If I were a model creator, to get some hype, I'd make sure that the model I'm training passes those "tests."
I have a suspicion that the snake game was in their training dataset, that's why it always nails it.
very likely.... most of the code gen llms so far have been disappointing... feels like code pairing with a junior fresh out of the uni...
id like to see it actually build a proper lambda script that pulls something ... transforms something and serves it... in an optimal manner...
or even better... a snake game but over telnet.... 14.4k style
It might very well be in the training dataset... but it passed almost every test thrown at it... even the "what does this code do" test... which I found to be very impressive. As these models get better and better we need to expand the testing to see if it can handle variations. But... this is so impressive. I don't recall any of the other models that Matthew has tested as doing this well before (even if we take snake out of the equation).
Thanks for this deepseek video.
I have tested it and it really blows my mind.
A copilot video will be highly appreciated.
As a programmer with 20+ years of experience I can say that now I'm starting to become impressed. The code (in this case quick sort) identification element is extremely valuable on its own. Finally an local AI model has as much smarts as a high schooler.
Running this now on an RTX4090. It works but damn it's slow. So "local" may not be the right word yet.
@@ArianeQube Do you know what the bottleneck is? Are you sure it is your gpu?
As a programmer with 16+ years of experience I can say that you sound like you have been sleeping if you think it wasn't interesting before this.
@@foobars3816 As an anime watcher with 0 years of coding experience, I believe he was referring to the code, not artificial intelligence.
It's true that Machine Learning is impressive and it's been achieving amazing feats, but the code it was generating by itself meant nothing, at least not until now that it's making some code that can actually be used.
As a programmer with less than 1 year of experience i can say that this is not yet interesting for me.
You are amazing, one of the few channels that I have with all the notifications enabled. Of course, I would love that you make a video with the installation of local VSCode programming assistant. Thank you very much.
This was one of the better code testing processes I've seen you do on model's. I don't expect most of them to get things on the first try but after some encouragement.
It would be great to see creative and effective ways to use open source llm models to build useful applications.
Isn't it great to be astonished almost on a daily basis by the progression of AI ?
I will have to get this going sometime this weekend to see if it can help me finish up my Inventory management app.
One question: Say I wanted to use an LLM to make a local AI for use by my company and website.
Can you give an overview of what is required as far as training etc..?
I would love to see this... ❤
Would love copilot tutorial
Thank you for another interesting video. Please do make a video that implements a local copilot. I use Olama, Litellm, Conda, and VScode on my M1 Mac Mini. I've also installed Autogen and Langchain with this setup. So far, my only problem has been using Docker to run code from Autogen, but it runs fine on the CPU. IMO, you are working on the most important projects for helping the community march towards AGI.
This is the ONLY model aside from OAI 3.5 and 4 that will actually write good code, python, js, bash, or whatever I have tried. Great Video!
I test this model to beat leetcode. It works like a beast, almost every coding challange is solved even for the hard problems. I guess we have a practical coding model already. Even better, it's open source.
Finally... this is awesome. Thank you for reviewing this model.
Yes I need to know how to run this model in vscode on local
Vscode tutorial of course - but would like to see the code suggestion/completion feature as well in it
Very cool :) Can you make a few videos about AI code assistants for C++ ?
Definitely want to see how you set this up. I wonder if you can ask it to increase the size of the tiles in Snake game without breaking? Aider couldn't do that.
I would be really interested in seeing a video on how to set up this model in VSCode!
My favorite moment by the end of the week when I discover new impressive project with your video 😊.
Lovit
Really happy that you changed the way you test the coding models, it really hurt me to see you only ask one very basic coding question that every model was getting right and then a super complex question that no model was getting right. With this new format the improvements are more visible. Also would love to see a way to use this as an AI code completion like copilot. (Maybe find a way to use it in Visual Studio as well as Visual Studio code, I've found some projects for VSC code but none for VS :( )
Yes, please. A tutorial for LLM in VS Code would be great.
Really appreciate everything you do. Thank you!! Tip delivered!
Tutorial +Pointer App, thank you
What's +pointer?
@@matthew_berman your highlighted mouse pointer
We need a 70b coding model!!!! It would crush GPT-4
Let's get that tutorial, save us some time haha. Love your work Matthew, thanks for your contributions to the AI space.
Bro how you said that the background is ugly lol😅😂😂 which is true.
And that’s the thing I like about you is that you are true to the core, brutally honest. Who can agree
Maybe it's good on basic, widely known code like snake. But it fails on something more complex which wasn't included in dataset I guess.
Definately the best coder!
First open source LLM into awesome level
Amazing. Super duper keen to see how to integrate this as a copilot substitute in VS code!
I love the direction this is going 🙂
Thanks for sharing 🙂
Could you test the models on strongly types languages instead of dynamically typed languages like Python and JavaScript ?
The fact that AI is becoming a common topic between the United States and China highlights some fundamental interests between both Nations. We both want to pursue technologies and the betterment of our people. This should be a technological space that foreign Nations can find common ground on. Maybe AI can become the platform from which we as a global society interact within an open discourse.
Very good, Sir. Thanks for the video.
autogen expert when?!!!
Lol...working on it!!
9:38 - it's just a 'font ligature' feature. working with a special font and modern editors
I use VSCode and would love a free copilot tool to help me out. Yes please to the tutorial!
The first model to create the Snake Game !
Please I would LOVE to see this tutorial in how to do It to run it on local. Thanks a lot for your great work and aim to share knowledge with everybody.
just did a quick test of claude 3.5 opus with a spin on one of your questions. I asked it:
imagine you put a ball in a cup and turned the cup upside down on the table. you then lifted the cup. where would the ball be?
Claude responded:
Let's think through this step-by-step:
You start with a ball and a cup.
You put the ball in the cup.
You turn the cup upside down on the table.
You lift the cup.
Now, to determine where the ball would be, we need to consider the effects of gravity and the physical interactions between the ball, cup, and table:
When you turn the cup upside down, gravity would cause the ball to fall towards the opening of the cup. As the cup is placed on the table, the ball would come to rest on the table's surface, contained within the inverted cup.
When you lift the cup, you're removing the container that was holding the ball in place.
Therefore, after lifting the cup, the ball would be sitting on the table, in the spot where the cup had been placed.
The ball wouldn't stick to the bottom of the cup or float in the air, as gravity would keep it on the table's surface once the cup is removed.
chatgpt 4o basically gave the same answer.
I’d love to see a tutorial explaining how to get this working within VS Code.
Your own free local copilot? Hell yeah, that would be interesting!
This year: Snake
Next year: Optimizing Crysis to run on a raspberry pi
That'd be great! A tutorial on using it with VS Code
Dude, i can't believe you have to ask if we want to know how to hook this up to VS code. This would be next level
Vs Code Tutorial would be really great
When are we going to get LLMs for other roles beside coding? I want a product owner, business analyst, software architect, and QA.
It would be great if you could briefly discuss the licenses these adopt and the type of hardware you’d need depending on the parameter sizes.
I run the 6.7B on 3090 using ollama and it's super fast.
It's performance is almost indistinguishable from gpt3.5.
Yeah, a tutorial for how to install an LLM as a copilot alternative would be fantastic.
hi, pls let us know how to setup the LLM in visual studio as a replacement to copilot. thanks.
My first game on a homecomputer was Lemonade on an Apple II. Many moons have passed since then...............
Mine was Zaxxon.
Yes please! Can we please have a tutorial for DS integration in VS or any other IDE. Thanks!
Yes please do the local coder tutorial!
Openchat 3.5 or DeepSeek Coder?
This is really handy.
Good stuff as per usual. This one hits all my needs... So let's get this.into autogen and see what kind of damage we can do.
When running the aider benchmark tests it also scored very well
it’s a slept on model. Competes very well against the GPT4 coders for sure
Interesting that it used the unittest framework which is pretty much secondary to pytest these days. I would ask it specifically to use pytest. I had this problem all the time when generating python code with tests. It’s also very interesting to generate test fixtures as well with parameters to make really good unit tests. Pytest is awesome.
Thanks!
Thank you!!
Yeah! Copilot-TUT VS Code, macOS! ❤❤❤❤❤❤
this is AWESOME!
You’re doing the copilot thing right? :) just got a new laptop with 48gb of ram and I’d love a free , local copilot (especially if it is half decent at rust)
Edit: actually I did that myself with continue extension and ollama as host :)
I love your code model testing!!
Awesome!
I want this as an API.
Do that vs studio guide. It sounds great, I will definitely watch it.
I'm new here. I have recently started watching your videos. It's would be really helpful if you can link the video to cards instead of just saying "the previous videos" 2:06 . There are too many videos on the channel and I really get lost in them.
Ok! I try to link them in the description but I can start adding cards.
When are you gonna release the llm finetune vid?
Love this!
make a mouse and keyboard macro recorder that has a replay function this would be cool test.
Can you elaborate?
@@matthew_berman Just something that can record the position of the cursor as well as any inputs, that can maybe be saved to a file and replayed. A not too hard but a bit unusual so it would be a better test to make sure it didn't just learn to copy.
Deep seek inf loops so much!
Time for a harder game. Space Invaders? Wonder if they are training specifically for your tests.
Tried 7b locally, impressive!
I want to see an integration with visual studio :D
Dude, give me that tutorial on how to set this up as my own copilot in VSCode!!
Please do the tutorial!!!
Hi Matthew, fantastic stuff, please do make a tutorial for this DeepSeek Coder AI in VS Code.
A tutorial will be highly appreciated, thanks
Wild! Loved it!
The gap between opensource LLMs and close ones is closing super fast. I'm wondering if there will be a day that we will get like an AGI which is based on an open source model. But with the pace and the "secret sauce" OpenAi is running, they are still keeping a large pace between them and the rest of the world.
Very impressive !!!
VS Code Copilot Tutorial would be helpful!