I'm still really underwhelmed by anything that is offered in vs code. My favorite by far is aider chat. It builds and entire map of your repo with ctags, and gives you the control to add exactly what files you want to context and uses a new unified diff editing structure backed by benchmarks which none of these other tools even come close to offering. It's completely terminal based and so I use it with neovim seamlessly. You can use it with gpt3.5 - 4turbo and local open source models (another advantage).
Thank you so much for doing this comparison!! I do have to say that there *is* a difference between *bugs* and *poor practices* - AI helpers work well with occasional explanations and are good at working with a hammer. There are many surface connections in life that, have a far superior answers like glue, screws, nuts-and-bolts. But given enough nails and enough popsicle sticks, one *can* use a hammer to build the Taj Mahal.
3:42 Did your computer have at least 64gb of RAM, no other processes running, and have any other household electrical devices unplugged so they don't compete for electricity? This is JetBrains we're talking about after all 😛
Hasn't anyone ever noticed how Jack and the popular American astrophysicist Neil deGrasse Tyson speak exactly alike? Please check one of his videos to confirm this. Two forces to reckon with in their respective chosen life paths. They also both look like they are in the same age bracket. Amazing!
Cody doesn't work much does it? I asked how many public methods a class has, it replied with 6 (wrong total). In the list one of the 6 was private, I asked it if it was sure that it was really public, it said my mistake, it's private, there are 5 total. I tested again saying that "isn't ToString also private?" and he said you're absolutely right, it's also private, so there are 4 total.
@@jherr if i may ask, why do you feel that these are questions you might not necessarily ask to such tools? If they can't tell a difference between a public and private method how can i expect them to find out more subtle bugs like the sorting in place? Plus the lack of conviction in their own replies is alarming, I simply asked if it was sure that tostring was really public and not private and it started hallucinating...
@@RobertoFabrizi Personally, I'm not looking at the AI as a source of definitive answers about something like the number of methods in a class. If I legitimately wanted to know the answer to that I would use a parser and look at the AST (and even then it's not so simple with inheritance). AIs produce a set of values with confidence scores. So with your question it probably generated several alternative answers to the number of methods both public and private and it had the highest confidence in what it presented as the answer. That's not a mechanism I would rely on to get "truth". Same reason that I would never take the code produced by an LLM at face value. I always inspect the output and make modifications
I don't think you understand how much computer power and storage is required for GPT- or similar model to run , even a single command.Unless you own a supercomputer in your basement, it's not happening
Why TF everyone include tabnine ? Tabnine is anything except AI. It has an early mover advantage but its suggestions are doog poop. It was a bad tool when no other tools were there and now it's worse.
Codeium for completions and the Phind (sometimes Perplexity) for searching.
Phind works beautifully, better than gpt4 for coding questions, that's for sure
Thanks for another year of great content, Jack. You're the senior/lead we all wish we had!
Thank you! I really appreciate the support.
Copilot got /workspace as well!
Phind pretty good as well for chat. I really like Cody for web for being able to understand a web repo quickly without having to open it in VSC
I'm still really underwhelmed by anything that is offered in vs code. My favorite by far is aider chat. It builds and entire map of your repo with ctags, and gives you the control to add exactly what files you want to context and uses a new unified diff editing structure backed by benchmarks which none of these other tools even come close to offering. It's completely terminal based and so I use it with neovim seamlessly. You can use it with gpt3.5 - 4turbo and local open source models (another advantage).
Did you manage to run aider via docker?
Thanks for sharing! Save me some time evaluating all of them
Jack, you missed Codeium!! I love their business model, and my colleagues and I prefer it to Copilot in many ways. Codeium is pretty dang good.
Try SuperMaven?
Always straight to the point in simple and fun way ❤
I believe Copilot has more context with @workspace agent.
Thank you so much for doing this comparison!!
I do have to say that there *is* a difference between *bugs* and *poor practices* - AI helpers work well with occasional explanations and are good at working with a hammer. There are many surface connections in life that, have a far superior answers like glue, screws, nuts-and-bolts. But given enough nails and enough popsicle sticks, one *can* use a hammer to build the Taj Mahal.
Codeium(simple completion) + Pieces OS(Code Context and Chat)
i've use cody s for a couple months now, it's great and it's free
Did you try using the /workspace command at 5:05 ?
I did and, weirdly, it decided that the graphql schema file was relevant to the query but then it hallucinated the schema again.
3:42 Did your computer have at least 64gb of RAM, no other processes running, and have any other household electrical devices unplugged so they don't compete for electricity? This is JetBrains we're talking about after all 😛
My favorite AI coding tool is Jack Herrington
As other commenters have suggested, does copilot find the schema with /workspace? That would really put it a cut above the rest
copilot -insiders at least as of just before the end of 2023 /workspace feature fwiw
What about `Code GPT: Chat & AI Agents` extension for VScode ?
Hasn't anyone ever noticed how Jack and the popular American astrophysicist Neil deGrasse Tyson speak exactly alike? Please check one of his videos to confirm this. Two forces to reckon with in their respective chosen life paths. They also both look like they are in the same age bracket. Amazing!
Wow. Thanks! That's a really nice compliment.
I really wonder how Codeium fits in the ranking! I've tried Cody but I find Codeium slightly simpler.
Me as well! I found CodiumAI to be *the* best AI code assistant out there! Shame that it's missing from the comparison.
How about open source models like mistrel codellama ?
I believe copilot looked at all files in open tabs. Might be worth trying your gql test again with the schema open in a separate tab.
Looking at your other replies I see that it also hallucinated with @workspace so I guess this is unlikely to make a difference.
I test Codeium, but no clear answer with TS, but good autocomplete)
Should have tried Onboard AI.
wnat about codeium ?
Codeium is good. I've been using it recently, up until Copilot released their extensions stuff.
Cody doesn't work much does it? I asked how many public methods a class has, it replied with 6 (wrong total). In the list one of the 6 was private, I asked it if it was sure that it was really public, it said my mistake, it's private, there are 5 total. I tested again saying that "isn't ToString also private?" and he said you're absolutely right, it's also private, so there are 4 total.
Hmmm, that’s not the kind of question I would think to ask. But I would test it against a few AIs just to see if others get it right.
@@jherr if i may ask, why do you feel that these are questions you might not necessarily ask to such tools? If they can't tell a difference between a public and private method how can i expect them to find out more subtle bugs like the sorting in place? Plus the lack of conviction in their own replies is alarming, I simply asked if it was sure that tostring was really public and not private and it started hallucinating...
@@RobertoFabrizi Personally, I'm not looking at the AI as a source of definitive answers about something like the number of methods in a class. If I legitimately wanted to know the answer to that I would use a parser and look at the AST (and even then it's not so simple with inheritance).
AIs produce a set of values with confidence scores. So with your question it probably generated several alternative answers to the number of methods both public and private and it had the highest confidence in what it presented as the answer. That's not a mechanism I would rely on to get "truth".
Same reason that I would never take the code produced by an LLM at face value. I always inspect the output and make modifications
Haven't got round to watching yet, but thought tabnine was *terrible*. I used codium and find it slightly better. But not much
Lol, just watched 🤣
I'd like an offline only version
I don't think you understand how much computer power and storage is required for GPT- or similar model to run , even a single command.Unless you own a supercomputer in your basement, it's not happening
@@coherentpanda7115you can run text completion stuff on notebooks, it works and obviously it's not comparable to gpt4, but it's as good as gpt3.5
Why TF everyone include tabnine ?
Tabnine is anything except AI.
It has an early mover advantage but its suggestions are doog poop.
It was a bad tool when no other tools were there and now it's worse.
@jackherrington did you write it like "@workspace write the graphql.."?
This should search and find the needed context.
I did and it found the schema but then hallucinated the query.