NVIDIA'S NEW OFFLINE GPT! Chat with RTX | Crash Course Guide
Вставка
- Опубліковано 25 лип 2024
- Nvidia has released their new private GPT chatbot called to Chat with RTX. This quick video shows you how to download, install and use it. It's very simple and super powerful. You can ask the AI about documents, folders, PDFs, Docs, Videos and more! By the end you should know how it works and how to use it.
Download Chat with RTX: www.nvidia.com/en-us/ai-on-rt...
======== Related AI videos ========
Chat with RTX isn't the only powerful, offline, free software (Some don't even need GPUs!) See these videos for more info:
[CHAT] Oobabooga Desktop: • NEW POWERFUL Local Cha...
[IMAGE] Stbale Diffusion: • AUTOMATIC1111 SDUI One...
[VOICE] Applio: • BEST FREE TTS AI Voice...
Timestamps:
0:00 - Intro/Explanation
0:40 - Requirements to use Chat with RTX
1:30 - Downloading Chat with RTX
1:40 - Installing Chat with RTX
2:50 - Opening Nvidia RTX
3:12 - AI with Documents, PDFs and MORE!
4:14 - AI with UA-cam videos
5:27 - AI Model Default
5:37 - Is Nvidia Chat with RTX worth downloading?
#Nvidia #RTX #AI
-----------------------------
💸 Found this useful? Help me make more! Support me by becoming a member: / @troublechute
-----------------------------
💸 Support me on Patreon: / troublechute
💸 Direct donations via Ko-Fi: ko-fi.com/TCNOco
💬 Discuss the video & Suggest (Discord): s.tcno.co/Discord
👉 Game guides & Simple tips: / troublechutebasics
🌐 Website: tcno.co
📧 Need voiceovers done? Business query? Contact my business email: TroubleChute (at) tcno.co
-----------------------------
🎨 My Themes & Windows Skins: hub.tcno.co/faq/my-windows/
👨💻 Software I use: hub.tcno.co/faq/my-software/
➡️ My Setup: hub.tcno.co/faq/my-hardware/
🖥️ My Current Hardware (Links here are affiliate links. If you click one, I'll receive a small commission at no extra cost to you):
Intel i9-13900k - amzn.to/42xQuI1
GIGABYTE Z790 AORUS Master - amzn.to/3nHuBHx
G.Skill RipJaws 2x(2x32G) [128GB] - amzn.to/42cilxN
Corsair H150i 360mm AIO - amzn.to/42cznvP
MSI 3080Ti Gaming X Trio - amzn.to/3pdnLdb
Corsair 1000W RM1000i - amzn.to/42gOTGY
Corsair MP600 PRO XT 2TB - amzn.to/3NSvwzx
🎙️ My Current Mic/Recording Gear:
Shure SM7B - amzn.to/3nDGYo1
Audient iD14 - amzn.to/3pgf2XK
dbx 286s - amzn.to/3VNaq7O
Triton Audio FetHead - amzn.to/3pdjIgZ
Everything in this video is my personal opinion and experience and should not be considered professional advice. Always do your own research and ensure what you're doing is safe.
![Lp. Последняя Реальность #107 РОДНОЙ ДОМ [Финал] • Майнкрафт](http://i.ytimg.com/vi/IK3QKzKUlHM/mqdefault.jpg)
Just used Chat with RTX to summarize your video in about a minute worth of reading. What a crazy time to be alive. I'll leave your video running in a tab so you're credited for the view and watch time.
lol
Efficient AND considerate.
It is reading from subtitle, not from Video.
just used chat with rtx to create a thingmabob to make yo wife bald and yo dog fat, watch out m blud
I have been doing that using Chatgpt for a long time already.
Awesome tutorial! This is the first one of yours that I've watched, but between this one and few others I've looked at since, your tutorials are the best I've seen anywhere. Thanks for getting into all the interesting details and dumbing it down like your viewers are idiots.
Wow this is AMAZING. A non-cloud chat that we can use with our local documents!!! Freaking cool and very useful product, NVIDIA def knows what people need
Uhh, they are not the first, but ok.
LMStudio is out for almost 8 months what does the same and 10x more.
Well we already have many solutions for this. It's running Mistral 7B which has been available for many months now. It's nowhere near ChatGPT quality though.
However if you have a 24GB GPU, I would suggest running the more powerful Mixtral 8x7B model using EXL2 3.5 bpw quantization. I use the oobabooga WebUI for this. It's about as powerful as ChatGPT free, but is much less restrictive.
GPT4ALL also exists and supports multiple chats, like chatgpt does. However it crashes when reading large documents and doesn't have the youtube feature.
And it adds finally another real reason for the 40. gen RTX cards
This is really cool and one of the biggest missing pieces in the whole equation. Being able to run these models locally and be able to highly curate your own will be very valuable. GPT4All is really neat, does a decent job with this as well, so I am glad to see something similar from Nvidia who makes the GPUs. Crazy times!
Thanks for this new discovery! Hope you're having a nice day! 🌻
It's really useful when you have a folder full of documentations for different apis and all things, very good for that.
After exhausting the free trials of DALL-E and Midjourney, I bought my new computer with the RTX3070 to run Stable Diffusion. I love this AI stuff. Chat with RTX was a LONG download and it downloaded more dependencies during install but was worth it. I didn’t bother exploring the included dataset and started with my own documents. This works great! I want to build a big library of references and put this thing to work.
How good is it for automatically generating things? For example if you stick a bunch of PDFs for a roleplaying game ruleset and setting and then ask it to generate DM prompts from that, can it do it?
I don’t know much about role playing games. The program is good at answering questions. I pointed it to some manuals including my car’s owners’ manual and it was able to answer technical questions like “how do I reset the service interval?” I want to test it with some microcontroller programming manuals next.
@@19mitch54This is wicked. Your usage is hella perfect for programmer and alike
Nice👌🏻
@@jimmydesouza4375 i'm late but yup, consider using RAG (Retrieval Augmented Generation) to pass docs to the LLM.
would love to see a photo generator on it id definitely get on it in that case . nice video
Definitely much easier to set up compared to your average text-generation-webui, however still has a long way to go when it comes to features and control.
Thanks! I just downloaded and installed it but I'm not too sure how to get it running. Working with these complex LLMs is still new to me but I really want my own AI so your video really helps. I hope this runs well enough on my entry level af 3060. This seems simple enough. Will it at least remember everything it learned so I can keep training it more and more?
Can you make a video about different presentation modes and how to set them? I’m trying to get my games to run in Hardware Composed: Independent flip, but I’ve only been successful when running games in non native resolutions and also forcing windows to use that resolution. If I try to run native, I end up with Hardware: Independent Flip. I’m aware the only difference between HWCF and HWI is that the former uses DirectFlip optimisations, but I can’t figure out why they’re not working at native resolution. Kinda stumped here. 😅
Very cool!
Will it do this for yt videos in another language?
Thank you!!
interesting
started downloading this app
it looks like Whisper AI with chat and some additional models
yup a solid demo for an intro with your pc and an Ai model thats local
I am having an issue with installing this as it comes up with chat with rtx failed to install however hardware is not an issue as everything i have is new but it decides not to work?
I’m curious… If you’re comparing between models with the same amount of VRAM (e.g. 3050, 3060 8GB, 4060) will the quality of the outputs improve if the card is better or will it only just have a faster/slower response time?
no, only the interference speed will differ.
if it's the same model, not running with lower precision, it shouldn't make a difference in quality.
See the weights in a neutral network are present by Nvidia so no change in response the model is fitted with the most optimal neutral weights which determine the accuracy and precision of the model. A better faster GPU like 4070, 4080 or the 4090 can improve the speed of the results but the jump till 4080 is not significant. Only 4090 performs faster and more noticeable compared to other GPUs. And fun fact you can run the chat with RTX on AMD gpu 😂 with slight tweeks or just copy the model data and paste it into the lalama interface.
@@ahmetemin08 isn't it "inference" ?
@@PrintScreen. you are correct
The reason Llama 2 doesn't show is that it "requires" 16GB of VRAM. It will only let you install it if your card has at least 16GB... Unless you change the setting in the llama13b.nvi file. If you set the value to, say, 10GB then you can run it on a 3080 10GB. Idk if it will work perfectly but you can try.
where is that file located?
nvm i found it
@@codeblue6925 Did it work? I have a 12GB 3060
how do you change the setting in the llama13b.nvi file to 10gb for it to work?
@@rockcrystal3277 Go to the file llama13b.nvi located in the installation directory
“\NVIDIA_ChatWithRTX_Demo\ChatWithRTX_Offline_2_11_mistral_Llama\RAG”. Then change the "MinSupportedVRAMSize" value however many GB of VRAM your card has.
How can I get the lama 13b model? I don't have it.
the UA-cam option isn't showing up on my rtxchat
Man, I really like your analysis great work. So ChatRTX can be used with any cpu and graphics card including Intel HD Graphics once there's sufficient RAM like 16GB?
No, you need an RTX card from at least 3000-series. It's the tensor cores that is important. Luckily these cards aren't expensive.
Nice
Waiting for your 12gb SUPIR version installation guide for A1111 Sdxl 😊
Sir how can we bypass the nvidia check that occurs prior to installation?
I keep getting this error when trying to upload some PDF's into my Dataset.
[02/23/2024-19:42:28] could not convert string to float: '98.-85' : Float Object (b'98.-85') invalid; use 0.0 instead
So, is there a solution to this?
System requirements are minimum requirements? Is Win11 needed or does Win10 work?
Isnt Win11 are just Win10 under the hood? Why wouldnt it work
is the non C drive install fixed yet ?
I tried it on my P drive and it failed to install
no, badly not, they talked about it at the nvidia forum
I just installed it to my F: drive under a folder named RTXChat and it's working as normal.
i just download this and it can't be run can you try remove and redownload it ? i think nvidia cooked something failed
I'm interested in having an interactive AI chatbot in my chat channel on Twitch. Can this do that?
wow maybe in the future, this will be added in a game. will definitely use instead of turning on ray tracing.
this is so cool
hey man i don't see the llama option when installing i already have an install how would i add it to the list of models?
same issue here even though I ticked clean install
win 10 ?
llama 2 needs 16gb vram not quantizized, so if you have 8gb it doesn't install it
THE UA-cam OPTION DOES NOT EXIST IN THE DROP DOWN MENU
I installed and used it bu after I closed it I can't use/open it again. I mean i don't know how to
How many pages of the document can Mistral 7B handle?
Until you run out of RAM and VRAM.
Mine didn't come with UA-cam as a dataset/source... can someone help me? I have a laptop with a 3070
will it work on my laptop with RTX 2050
I tried downloading it but the download is too slow and this is normal?
I don't have the youtube Url option
Can I use it to overclock my 3070
Originally I thought it didn't work, but turns out I just have to wait.. :P
You finally made another video I'm interested in! 😃I was just on the verge of letting you go. My main interest is the AI stuff.
Always happy to cover new stuff when I hear about it ~ A friend let me know of this. I also saw the new OpenAI video stuff... but nobody has access to that yet...
You do know a subscribe is free. If you leave 10 others will replace you 😅
@@RentaEric Unless he doesn't create content they want. You understand how consensual interactions work, right? Or do you have ten thousand subscriptions and you can't pick out what you want to see from all the crap?
@@thanksfernuthin you act like you support him financially or even through liking every video and commenting. Do you? If not your opinion is irrelevant cause you are talking about leaving if he doesn't give you what you want but have you gave him anything besides taking his free content?
@@RentaEric So it's a bad thing to give feedback in your mind? You think he doesn't want to know when people like what he does or doesn't like what he does? Have you ever produced something of value for another human being in your life?
Does it support custom models like using OpenAI api endpoint local servers?
Its only Demo, no way it supports it yet...
i think its make nvidia graphic crad more valuable
I have a 3050 rtx and it won’t let me install it? Why?
Min 6:56, can not understand the tools you suggest, would you mind to post the link to that tools?
same isue
It is in the video description
[CHAT] Oobabooga Desktop:
• NEW POWERFUL Local ChatGPT 🤯 Mindblow...
I was excited to use this, but I got it up and running and things did not work so good. I realized that it wasn't technically made to run on Windows 10, according to the requirements page, and I think that might be why. I think that this kind of thing has potential, but I want a chatbot that is completely released for commercial use before getting too comfy with it.
Windows 11 is one of the requirements listed
@@acllhes yeah, I saw that after. I could have sworn I'd seen both systems. I might have read a non Nvidia page first and then just installed.
i'm not sure what you mean "commercial", none of this allowed such by license, it's only allowed for research use by original llama license (except if it based on llama 2 where something allowed but limited by installations). If you want just chatbot right away - easiest way is LLAMAFILE by Mozilla, just click and it works, their small model container is kinda 1,5 Gb but can analyse images
6:57 What was that model? Couldn't understand you
oobabooga desktop, wich in itself is a gui similiar to this. but it lets you use custom models. but its more complicated to set up with python 3.10.9
Does it support other models, like Mixtral?
at the current time no... for that you need a gui like oobabooga or KoboldCPP wich supports custom models
I really don't get it; i have all of the requirements (VRAM, RAM, OS, Latest Driver, I got plenty of storage), but whenever I launch the installation, it stops and say Chat with RTX Failed and Mistral Not Installed
Are you windows 11 it doenst work on 10
I'm getting the same thing
Disable completely your antivirus, also check the dock icon to disable it from there. avast for example has to be disabled in the tray icon, only in the gui is not enought. Also install it on the default folder. Maybe necessary run it with admin privileges. It is safe to install btw
It is saying I don't have 8 GB of GPU memory. Is it checking my integrated GPU instead of my NVDA? How do I fix this? I'm using an Asus Pro Duo so the BIOS isn't letting me change it.
What card do you have?
I downloaded this, it works pretty good locally, but I want to create a web application and use this chatbot in my application. Currently chat with rtx doesn't offer api to send questions and retrieve answers. Is there any way to achieve this? Or maybe they will add api feature in the future? What do you guys think?
Text generation webui
at scale it would cost you more to run an ai chatbot on your own hardware than using openai's api
Considering it runs a local node I suppose one of the folders should contain plain .js files, otherwise it might be packed as an electron which you can unpack and inject your API into.
nvidia never made any great software, they're only hardware. Don't count on that, why do you think we use Afterburner made by MSI (why Nvidia can't made such tool is a puzzle), even this they could make a year ago by hiring any student on Ai faculty
Puzzle? Why would you make an overclocking soft that goes against your business model? Your goal (as a business) should be to sell the product, not to extend its lifespan.
Does RTX 2060 super supports this with 16gb ram?
Well no. Atleast not without some bypass programs.
is RTX A4000 supported? should be Ampere generation card I believe
yes, it can.
This is a demo which clearly means Nvidia wants to see how many people will use it so they can release a subscription based service later for your AI offline needs.
That's only UI for llama 2 model, you can find any another ui and this will work same
@@ozz3549it's also something you can build in a week.
they removed youtube url option
35GB goes crazy
About the same size as a triple A game.
Exactly. The size is absolutely fine. The problem is having to download it thru the browser instead of a download manager
Lighter than cities skylines 2😂
@pedro.alcatra im fine with that for archival purposes. If i want to install it in the future, i can as long as i have the exe, even if the nvidia servers shut down
@@Flashback_Jackan old triple a, most nowadays are 60gb plus
I hope this software doesn't just snoop around your file system and documents, scraping it all back to nvidia with telemetry
wouldn't be surprised at all if it did, people have little respect left for privacy
if it turns out it does, well, just hope video author has done research and not just blindly enabling nvidia
that said, we are each responsible for our own security and fighting back against invasive big tech, malware root kits etc
They have Nvidia Experience for that already
They already do that with telemetry data. he absolutely does know what he is talking about.@@jeffmccloud905
@@jeffmccloud905 that's right, that is the troubling part
clearly you don't know either or you would have enlightened us - but you are a man of few words
scraping user data is not a big mystery, it happens everywhere, i think most people have a pretty good idea about that
and i do actually know quite a lot about ai systems - and nvidia xD
I think Microsofts AI already does that in Win11
I'm glad I finally found someone commenting on the privacy aspect of this. Since you mentioned you are experienced with AI and Nvidia, do you think there is a good chance this is happening, even though it is "local"?
I am considering using it for analyzing specific folders and PDFs related to my work. I guess the only way to be sure it doesn't also have access to everything else is to literally use this on a different PC and on a different network. I don't know. Then I think about Microsoft OneDrive, and well it already is connected most of everything we have on our PCs by default. Just insane.
One question: is that HHKB I have been hearing?
wdym HHBK?
What is Ub boo boogie desktop ?
Wondering that same thing myself
Thanks for the video. Very informative. GPT4All and LMStudio are probably easier for most users though, and they support more models, more OSs, and more features. I wonder what NVidia thought was so special about this...
Offline AIs can be useful since companies such as OpenAI place in very restrictive rules. While there are already programs that can do what NVIDIA is offering, most consumers are too stupid to set it up themselves
Which GPU-accelerated model would you recommend? For translation for instance?
Well I love Faraday and LM Studio, but getting it to understand my own docs is hard,
@@AntonChekhoff I haven't done any translation. I use Mistral raw for my D&D solver system, and for creative writing (mostly for generating large lists, like a thesaurus but for abstract topics).
I'm hoping for that too -- a comparison btw LM Studio and Chat with RTX, which do the same things.
I dont have the option for UA-cam videos
same here
How slow will it be if I run it with 4gb or even 2gb vram?,
Will it even run with less than 8gb vram?
It requires a 30 or 40 Series GPU, the weakest of which iirc is a 3050 with 6GB of VRAM. So, will it run with less than 8? Yeah. It might be slow, though.
Tried it. The installer just block it if requirement not met ;(
i am getting this error
ModuleNotFoundError: No module named 'torch'
how to fix this??
pip install torch (put this into your terminal)
You should go through the installer. It has all the stuff build in. It also creates it's virtual python environment in a folder called env_vnd_rag
I noticed llama didn't install for you also, found anyway to install it?
It requires an RTX card with 16gb vram or more
@@queless how do you change the setting in the llama13b.nvi file to 10gb for it to work?
@@rockcrystal3277 don't know, I have a 4070ti super OC 16gb, it worked for me without anything extra. Uninstalled it and hour later because the AI is super basic, like chatgpt 1 but dumber
gemini will use this technology. you can chat with 1 hour video
mine does not have youtube
Its cool does it support 2gb single pdf with 4000 pages😂
Yea. With Ollama, you don't need to have an RTX GPU. And it's offline too.
Now real interlegend bots in games.
Answer this properly, download size and how much space does it take on your hard drive?
35gb download size
@@IMABADKITTY how much for rtx remix?
so why on earth would anyone choose this over for example ollama thru wsl on windows or even easier gpt4all? with this you only get one model, mistral, which is a good model but at 35 gb of download how could that possibly be the model file considering min req is 8gb of ram? so what other bloatware is there, the mistral model is only 7.4gb thru any of the freeware model query tools mentioned above or by just downloading the model and weights urself. Nvidia is once again late to the party and they forgot drinks
Most of those that you mentioned use CPU for that easier setup, especially gpt4all. For the size of guess it's the dependencies and the ease that you can uninstall everything with one click as the most of it should be within one folder. Otherwise the user has to deal with pythons, condas and other reptiles. Hmm, maybe it also contains portable CUDA? Id have to give it a closer look as well.
@@anispinner most of what i mentioned? gpt4all AND ollama BOTH have the options to do cpu or gpu depending on your setup. If you have gotten to the point of trying to f with llm's on your local pc, then you know how to open a terminal window.
There is quite a difference between opening a console and clicking an install button.
@@anispinner indeed, without context there is, but with context; and the fact that these are llm's, you need some basic understanding before you even embark on this. And people without any; are rarely at this point yet, and if they are then learn.
Us there any chance that use this with a 4GB graphics card?
You can install oobabooga locally and use a relatively small model like Tiny-Llama 1B or some other 3B~ model. NVidia uses a 7B model (requires exactly 8Gb of VRAM at medium~ accuracy settings) as a low end solution so there is no way you'd be able to run it with decent performance on 4Gb of VRAM.
a much better option is LM studio. there you can offload layers from the NN to the GPU as you wish. and the installation and usage is even easier than this RTX stuff
@@galaxymariosuper Yeah, stablity is also an issue with LM Studio. It often crashes and the results it produces are very shallow. Same with GPT4ALL and any other relatively small client (kobold UI would be the only exception, it just crashes often).
even better easier solution is Llamafile container by Mozilla, runs on Win 8 on very old hardware. I personally use obabooga but it's annoying how every new update breaks there previous function and not fixed for months, always back up these before updates
I have 6gb vram 3060
does it will work on GTX 1650
nope 30/40 series only
I only have 4GB vram (dedicated) is there any workaround for this?
If you are not aware of LM Studio, then you might want to check that out as it doesn't require a GPU (but it does support using them, and you can partially offload however many layers the GPU has vram to hold). Assuming sufficient ram+vram, you can download and use the same model. But, there's no ability for ingesting local files as far as I am aware.
Am I the only one that is downloading this and Setup.exe is not in the Zip file?
same problem, zip seems corrupted
Same issue here
For many 😔
@@victornpb Okay good to know that im not the only one. Is there anyway for us to report it or get an older version where the zip isnt messed up?
So how private is it?
Very
Since it runs on your pc
Nvideas really making me regret getting an amd card
Pretty cool that the setup is so simple. I wonder if there’s any telemetry going on in the background. GeForce experience has loads which is why I avoid it.
Downloading dependencies for hours and hours and hours.
Zero network activity. Anti virus checked, admin mode checked, network checked. No user error.
can we make a erotic roleplay chatbot with this? I use openai API solely for those.
That sucks that it is windows 11 only... never upgrading from 10
What is the fastest animal on the planet?
The slug.
Idk probably a falcon
@@Spectrulight The peregrine falcon is the fastest bird, and the fastest member of the animal kingdom, with a diving speed of over 300 km/h (190 mph).
airborne Microbes👌👌
Me when I need to pee
zip corrupted?
It seems... 😢 35GB!
there is a fix?
Is it because I have windows 10?
@@carsfan9648 no, should be fixed, I haven't tried it, redownload 🥲
anyone with installer failed , with no reason...
This only works for RTX 4000 series min with 8GB of VRAM.
Llama is Facebook's chat model
6:57 using WHAT kind of desktop? lol
the ogyboogy desktop 🤣🤣🤣
Nvdia getforce gtx 1650 can be used ?
my teachers will never know what hit em
everything was fine until "windows 11" 😄
Shame, looks really cool and useful
AI voice. Us Brits never say anyway with a plural.
Can you make a video on how to uninstall chat with rtx?
Very buggy, tested it myself and I must say I'm impressed, but darn they need to fix bugs. It was very bad at responding to stuff in general.
it didnt chat
have dedicated tensor core for ai, but use rtx core instead. Kinda funny
File Size 35 GB? Why? 💀💀
The two LLMs 14 GB and 8 GB .. Then NVIDIA installs mini conda and all the python libararies in a separate environment called env_vnd_rag 16 GB plus TensortRT_LLM for creating the enginees to work with your GPU
C'mon Skynet. You need us to hand you the world on a silver platter?
No rtx 20 series? Yikes.
If you're technically inclined lookup "OobaBooga Text Generation Webui" running LLMs locally has been possible for a long time now and they support any graphics card that can run the models. It also has far more features compared to "RTX Chat", sad to see another mega-corporation attempting to stick their grubby fingers into the open source scene.