TINY 70 watt RTX 4000 SFF ADA Generation GPU For AI, Docker, Plex, Jellyfin, and MORE!
Вставка
- Опубліковано 4 чер 2024
- Newegg Link: newegg.io/nca0ed1a8c
Check out the Ubuntu/Ollama Forum thread here:
forum.level1techs.com/t/ubunt...
How could something so small be this big of a deal?! Well, Wendell is here to show you! Follow along with Wendell as he sets up his RTX 4000 SFF ADA generation GPU from Nividia for a streamlined AI experience, a home server beast, and whatever else your creative mind can come up with!
0:00 Introduction
1:26 Background and Use Case
2:52 VS the 4090?
4:06 RTX 4000 Specifics
5:00 Software Setup
6:12 Step by Step
17:12 Wendell Reflection Time
18:24 Results and Issues
28:24 Conclusions
********************************
Check us out online at the following places!
bio.link/level1techs
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.
-------------------------------------------------------------------------------------------------------------
Music: "Earth Bound" by Slynk
Edited by Autumn - Наука та технологія
1. Take the cover off the gpu,
2. plug the little fan into a fan extension cable and route cable out the cover as you reinstall the cover
3. plug extension cable into mobo after you put the card back in the computer
4. ~5 watts unlocked. 5/70 is over 7%, so you can get a nice little boost on the card's performance freeing up that bit of the power envelope
I read about someone doing this on the A2000 but I cant remember who it was to give credit
5. Poop on it
It's an older meme, sir, but it checks out
"kids these days just don't appreciate rubber hand puppets."
Only level 40+ Memers catch that reference
Timestamp?
He never addresses whether or not he actually pooped on it. The people deserve to know.
That's what she said.
4000 ADA! Literally my favorite GPU of the last few years. Glad you got one to play with finally 🙂
can't wait for the next Beer review at the end of your video 😁
7:16 Props to Wendell for calling out the incredible danger of curl piped into SH. Did you know, for example, that a server can often tell via side-channels if you just curl a command, or if you curl it into a shell? It's very possible to create a downloadable URL that delivers one script if you just curl to stdout/a file, and another if you curl pipe bash. Always download shell scripts, review them, *then* run them!
yeah but be very careful of what you run!
The easiest example would be the user agent header.
Yeah I do not like when websites give a curl to sh pipe as primary install method
@@fgregerfeaxcwfeffece Ah, almost! The user agent can tell you that somebody is using curl, but not if they are looking at the output! What I'm talking about here are things like if you pipe curl into a shell, the shell accepts inputs in a very specific pattern, which can be detected. It's not about detecting that you're using curl, it's about detecting the difference between `curl whatever | sh` and `curl whatever > output.sh` . One should never pipe curl to shell, even if you've verified the script before, since the script you have reviewed is not the script being run! Even something like `set -x` can be trivially worked around(e.g. ANSI escape code). Using `curl | sh` is literately giving the remote webserver complete shell access, and it knows if you're looking at the script or not!
I am willing to bet my eye balls, that barely anyone ever reviews what the homebrew setup script really does. I'd argue most users can't even tell. And that's just one example.
Just a minor point? The application 'nvtop' is not Nvidia top. It's 'Neat Videocard Top' as it works on all GPUs. Everyone just assumes it's nvidia top but it's not.
nvitop is far better.
btop is my go-to, but I like seeing all these other tools in case I'm missing something by using btop.
Nvtop does not seem to work with Intel iGPU according to the app. When it detects an Intel iGPU, it will throw a warning at the start.
Note that NVTop stands for "Neat Video TOP", not "NVidia TOP"
It works on AMD and NVIDIA, and recently got support for Intel!
@@adam.maqavoy Those two monopolies are exactly why AMD and ATI were allowed to merge
Aha! I was wondering about that. I only knew about nvidia-smi and thought it was the only option from Nvidia.
That's a relatively recent development. Iirc amd support only came late last year. ua-cam.com/video/ehm2hmk_p3E/v-deo.htmlfeature=shared the 1.0 release was top for nvidia gpus. But I'm glad it's evolved into neat video with support for more than NV
RTX 4000 ADA + Minisforum BD790i = the dreamiest ITX build combination.
I'd love to see what that combination would do.
@@heirtothethrone000 it's so tempting. If I wasn't on assignment in Tokyo for the next year I'd build it this summer. Computer parts are so hard to acquire here, and even when they are available the markup is insane.
^^^Current dream build. And a Noctua passive cooler.
I don't understand half of what wendell is saying...but my god, its well spoken, depp techtalk, and i'm a simple guy... i see a new wendell video, i click on it, i watch it
Got an A2000 16gb on the way for my trans-coding galore, can't wait to tinker with it. Wendell dropping appropriately timed videos as usual.
there's a big difference between A2000 and A2000 Ada btw
@@TazzSmk There is no 16gb A2000 except for ADA Gen :)
@@SuperMari026 So it's a franken card with the mobile GPU? I've got the 12GB desktop variant.
@@linuxpirate Pretty much. I got it for €600 which is similarly priced as the non-ada 12gb variant.
@@TazzSmk Which is why one is called "RTX A2000" and the other is called "RTX 2000 Ada."
I have the RTX A2000 12GB for similar purpose. It’s less performant in basically every regard but also much cheaper and can easily chill in a 2U chassis :)
That m.2 being held in by the pressure of the gpu made me chuckle :)
More like cringe.
Another interesting aspect of this lower power usage is that you could install several of them into a workstation without having to seek out specialist power supplies. Four of these would come to 80 gigs of VRAM at just 280 watts. Too bad they're so expensive.
As Wendall mentioned though, in Quadro land -- this is a bargain.
RTX 6000, when it launched, I think was on the order of like $10k USD EACH.
Something like that.
And if my memory serves, the original DGX1 system which had eight DGX GPUs, was priced at $150k USD.
So, this is 1% of that price.
PER CARD, it might have 1/3rd the compute resources, but you buy enough of these, the efficiency is significantly higher than the bigger (and more powerful/power hungry) cousins.
So unless your model NEEDS 80 GB of VRAM on a single card, having a BUNCH of 20 GB cards would be absolutely AWESOME.
You'll run out of PCIe slots (being that the card is double slot width) way faster than you'll run out of anything else.
Get one of the 4U or 5U 8-10 double-slot GPU server chassis and just drop a whole bunch of these in.
Would be a killer system.
Is there enough power budget to do that? To run multiple of theses w/o power connectors? I have no idea what total power the board might have available for each slot.
@@chrisspellman5952
"Is there enough power budget to do that? To run multiple of theses w/o power connectors? I have no idea what total power the board might have available for each slot."
Depends on the motherboard, but I mean, I would imagine that if a Threadripper motherboard can have 7 PCIe x16 (physical) and a mix of x16 and x8 (electrical) slots, presumably, if they build it, then the board needs to be able to supply that much power.
Plus the fact that a Threadripper, for example, can pull 300-400 W via the CPU socket, so I would imagine that a motherboard is already designed to be able to supply this much power to the things that are plugged into it.
@@chrisspellman5952 Motherboards are designed to supply 75 watts to every PHYSICAL x16 slot and 25 watts to every PHYSICAL x1/2/4/8 slot.
If a motherboard has >3 x16 physical slots, they have a PEG connector or other supplementary input for additional wattage.
@chrisspellman5952 Chris replied and is nearly right, to clarify a liitle: WS motherboards tend to have additional pcie power inputs on the board - this is the type of board that will power multiple with no issue. A consumer motherboard will Not work with 8 of these (even if you had the lanes, they do not provide enough power)
Woah! Super interesting card!
Since its a card with such a focus on power, I would've loved to also see some numbers on its idle consumption (e.g. for always on homesystems).
Still such an amazing video! Keep up the great work @ level1techs!
I help contribute to open-webui, and we recently refactored the Dockerfile and added the feature to include ollama using build args and build the container for GPU accelerated embedding. So you don't have to install Ollama in a separate container, unless you require that.
That’s handy, thanks!
For years I have asked about better graphics cards for imaging, design, 3D, etc. I use a lot of apps, and I do things that aren’t normally done with graphics. Whenever I ask around on forums, I’m either flamed or given vague answers, always boiling down to “just get a gaming GPU”.
THIS video touches on some of the stuff I’ve hoped someone, somewhere, would cover.
I love my macs, but I’ve wanted to build a powerful machine so I can do more iterations, of higher quality, as fast as I can think of them. Fitting in SFF and not using a ton of wattage would be great too.
Thank you! More content like this please!
Factor in the external watercooling I need for my dual 4090s, and I'm really regretting not having had this card as an option when I built :D
Damn that's a pretty potent little professional class GPU, might be the same price on paper as a 4090 but you can pretty much guarantee it will work with any recent Server or Workstation without any concerns for new components like a PSU or generating crazy amount or simply fitting in your existing systems/taking up multiple PCI-E slots or risers.
So in that sense it actually makes a lot of sense over something like a 4090.
I can only afford the RTX 2000 ADA. The RTX 4000 ADA seem to be a pretty good option. Good video!
i cant wait to pick this up in a few years for my plex box
Awesome video like always but you are just teasing us with $1500 hardware, For the next video please do a ARC 770 video showing how to enable SRIOV!
same thoughts
Yesssa
That was mentioned like 7 months ago. He never made a video about it?!
great video! I have been planning to build an easily nomadic egpu setup for running stable diffusion (with ComfyUI), doing pure inference, just image generation. Would you have any speed comparison on that use case vs a 4090... without ollama running at the same time? or would you say the performance in this video would apply to me? Thanks!
I’ve learned so much but watching these videos, I need to check out their forums
in 2 years this will be an awesome deal
I own one, for a small pc gaming build, its more powerful than my 2080ti its amazing.
And with DLSS3
I just built a system with a Threadripper 7960X and one of these cards. The software isn't all there yet, but I'm planning on running a bunch of VMs with XCP-ng including an Ollama instance. It's insane that you can get something like this for 70 watts, even if it is a bit expensive.
If you run into a W6800 (older card now but similar price) or hopefully one of the W7000 series cards it'd be interesting to see how the value proposition works out for those playing with Rocm
Time to get two for a home server. One for Plex, another for everything else.
virtualize plex..or run it in docker...then just 2 homelab servers, 1 for backup.
@@rezenclowd3 TrueNAS exists and uses graphics cards in the same fashion.
Thanks!
I don't understand any of the stuff going on but I believe Wendell when he says it's cool and awesome
The form factor is the big advantage here. If you want to build a nice little home server with inference capabilities, this card is a great choice.
Looks pretty cool. I've been looking for a new GPU for machine learning, but other than some Tesla P100's on eBay I've not found much. I want about 16GB or more of VRAM(preferably HBM2).
It's nice to see some non-gaming GPU discussion!
I'm very happy with my RTX A4500, 200W max, 20GB ram, etc.
I think it's often overlooked.
$1,250 not cheap !!!!!
Definitely not cheap
but still can't buy a 4090 (will be last gen card in less than a year)
Which it's kinda disturbing seeing this kind of price we have on non Quadro GPU right now
Compared to a $2k RTX 4090,, it is!!!
In Quadro land where the RTX 6000 launched at $10k USD, it is cheap.
And the original DGX1 system that Nvidia made, which had eight DGX GPUs in it, was priced at $150k USD.
So this is 1% of that price.
To that end, this is very cheap, relatively speaking.
An A580 is overkill for plex.
YOU DON'T NEED HIGH-END HARDWARE FOR YOUR HOME SERVER!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
@@Splarkszter
"YOU DON'T NEED HIGH-END HARDWARE FOR YOUR HOME SERVER!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
I find it amusing what you think people do or don't need, for their home server.
(This ASSUMES that you know what people are using/running on their said home server.)
Meanwhile, I'm trying to get SR-IOV up and running on my Mellanox ConnectX-4 dual port 100 Gbps Infiniband network card, so that I can containerise my HPC applications because IBM decided to kill CentOS.
(One CFD application is certified to migrate to Rocky Linux 9.3, one still needs at least CentOS 7.9.2009, and another uses Ubuntu 22.04 LTS.)
If you had a ton of surveillance cameras (a mall?) this would make for a great realtime tensor object detection tool
Very cool, but I agree with the cons also. This could be quite good for a multi-model scenario (e.g.. autogen). It is a potentially good sign for something better to come, also.
Fan sound output differences between the cards are an important factor too.
At 19:04, did you use the word "fácil" in place of "easy"?
20:22 -- no NV link, no vgpu . . . imagine if it had that, too ?! Level1Techs has been on fire 🔥 lately. Incredible content 👍😎👏
I was really looking at possibly a pair of these to build a personal geforce now + virtual AI workstation but at $1250 each it's a bit much for me to spend in my homelab. Then I considered a pair of the A2000 12G but for a price between them you can get a pair of 4060ti 16gb. At the moment I'm tempted to pull the trigger on the 4060ti route next payday unless I run into a reason to go another direction.
just get the 4060Ti... its a gem of a card. All of youtube, reddit shits on it but that card can do wonders. I use mine undervolted and it consumes ~90-100 watts max.
did the output of dolphincoder already run on the rtx 4000? If so, did you have anything else claiming GPU ressources at the same time? Reason I ask is that my 6900XT on ROCm 6 is way faster than that, which is surprising to me. Setup is ollama inside of a ubuntu 22.04 distrobox running on an ubuntu 22.04 host.
I think that this is probably THE most powerful SFF card, in existence, right now.
A used GV100 would be a little bit more than this, price-wise, but it also consumes 250 W vs. 70 W, and it's a FHFL card rather than a HHHL card, so it doesn't fit in a lot of shorter server chassis.
I think this is probably THE most powerful SFF card in existence right now.
Fixed.
@@tolpacourt
I am fairly confident that there are other SFF cards that aren't public, that are probably more powerful than this.
It's just not public.
(By the time this card hit the shelves, you can bet that Nvidia is already working on the next one that's going to be even more powerful than this.)
Cool product demo! I’ll have to pay closer attention to Quadro features and support as opposed to the gaming cards. Newer to ML, most recommend the 4070 Super it seems for a homelab, but limited to 12gb of RAM and 200W
Thank you so much for the comparison. I just need to find an A2000 comparison for AI now.
Almost dream specifications, needs to be single slot to fit into my server.
Been using the 12GB A2000 for a while now, its perfect for me so far.
This is truly a holy grail...
Great Video Wendell .... I wonder if it'll fit in a MS-01 ... Prob. too big ... but I can Dream 🙂
Do you have any tips for enabling ollama amd compute for the gpu integrated into AM5 chips? I've got a Ryzen 5 7600 in an asrock rack b650 motherboard running my proxmox homeserver with an ollama instance. It works fine running on cpu, but i'd be curious to compare the token generation rate between the iGpu and the Cpu.
AMD doesn't support integrated graphics for ROCm. In fact, there's a lot of conflicts(which is why they don't want to touch it).
People have gotten it working with actually good iGPUs(780M, not the bad Ryzen 5 7600 iGPU which is essentially just a video out driver), it's not useful at this point, introducing new bottlenecks in place of old and shifting "what is slow" somewhere else.
I could see this added to a NAS. Containerize here and there and you have a solid home lab
sorry i think i missed it but can't find it in the video.... how many transcodes can it handle?
Ist is possible to run two in parallel for inference? To use a LLM model > 20GB ?
I wonder if the SFF PNY version will fit in a Dell R30/R640? If it would, it could be amazing with Proxmox
That is an exciting card right there.
@bamtoday mentioned the BD790i in his comment and that's actually what my brain was revolving around except, how would this RTX 4000 SFF fare in encoding/transcoding against the Alveo MA35D? i am really hoping myself to be able to use that particular setup for two system streaming but i have yet to confirm software support for such a configuration. obviously there is already software support for NVENC so this card will work but should the MA35D get support, will it outperform this card in the intended arena?
I couldn't find a preorder for less than $1200 this generation versus the A2000 before it that was often available for well well below MSRP due to the Crypto bust.
the 4000 and 2000 are not on amazon so 🙂 where do you get it with international shipping
May you run these with or without gaming GPUs on the TRX50 Gigabyte with a 24-core thread-ripper please? I would love to see you stream a demanding game while running image stuff in the background
$2,269 AUD makes it in between the price of the RTX 4080 and RTX 4090, with the RTX 4090 outperforms RTX 4000 SFF Ada Generation by an impressive 88%.
I was very intrigued by this card, but then I saw the price, and I IMMEDIATELY lost all interest. Fuck you, Nvidia. I get that there's a premium on Quadro cards but give me a fucking break. Also, while I'm talking about this, bring back the Quadro branding already. Your idea to axe it was dumb as hell.
One consumes 70W, the other 600W, the math checks out.
You have to consider performance per watt. This thing runs on the PCIE slot without supplemental power, meaning no more than 74 watts. What is it going to cost you to run a 4090 versus 2 RTX 4000's that draw, together, less than one third the power to do the same task.
For professional work these things crush the 4090.
@@KenS1267 I really don't understand the performance/watt argument. The RTX4000 is 1/3rd performance and 1/5th the power usage of a RTX4090. That would be an interesting trade-off if it was half the price. But it's not.
(Unless you plan to run it non-stop for 8-9 years, then it will break even)
@@Ardren I agree, the cost of the hardware makes it useful only in circumstances where the power consumption is critical imo... otherwise the value prop for the 4090 is just better.
Question: How does the AMD W7900 (48gb) compare to the Nvidia RTX A6000 ADA (48gb) for a similar "Swerk-station" Local AI setup?
twerkstation
I was just talking about triupth the insult dog recently. Nice , regardless i Love these tyype of vids , dont need it , but why not? well done
The next thing that is going into my homelab rack is either a Dell R7910 or a Dell R7920. When the that decision is made, I want one of these cards in it.
Man I want one
@Level1techs possible to cover Alveo MA35D ? 32x AV1 streams at 35 watt for ~1599 usd ?
@level1techs will 3 of these run well on trx50 gigabyte?
How would this compare to any of the Intel graphics cards for a media server?
How is it running these GPUs in an external GPU case? Most times i don't need the power. But the days i do i would love to plug in and use it.
What kind of Jellyfin or Plex server do you run?
How are they with Folding@Home? :)
Looks interesting for an Iray renderbox.
Would love if Wendell , since he mentioned Plex, would test out an Intel Arc Pro A40 (AXX series), as it's the only card(s) I'm aware of that claims to support Dolby Vision... It seems the PC world has seemingly given up on Home Theatre... pretty much no Blu-ray decoding since the Intel Core Gen 10, no HDR10+ or Dolby Vision, and I think only one company makes a CEC device for PCs, you'd think by now since SBCs like the Pi have had CEC for 5 years, PC Motherboard makers would be trying to add this feature and the relevant hype to get their pound of flesh... so much for progress.. maybe the AI on the RTX 4000 can figure out a decent movie or entertainment
Can we get a 4070 version for around USD 600 less, but with native HDMI 2.1 ports?
Pretty please.
I hope they make a Blackwell rtx version of this. With a little less vram and without ecc so it’s more affordable. And then obviously a Quadro version too. There would be quite a few gamers that would want this for ssf, home lab users, and I bet some system integrators would make some cool systems too.
Stable diffusion can easily use more VRAM if you are generating multiple images in parallel (use batch generation).
I'm thinking of buying a W6800 rather than this RTX 4000 ADA as they are pretty close in pricing where I live.
However, I don't know if that's going to be a useful purchase given that this is a bit of an older card too.
W6800 PRO has 32 GB VRAM vs the 20 GB VRAM on the RTX 4000 ADA.
Any thoughts about this?
I'm still trying to decide if the mispronunciation of "epitome" was deliberate or not? :)
Real question but kinda dumb.
Can i put x2 of these in a build and run a GPU per game, sure might have issues here and there but if its doable, sounds kinda neat for my use case, i have black desert often running and do have another GPU dedicated to play another game at the same time would be insane for my silly ideas.
What a brain, thanks for the info, Cheers Terence
This card potentially has a huge market for enterprises that want to upgrade their non-GPU servers to add AI inference support. There's tons of Dell/HPE/etc servers out there that have 75W PCIe slots, but no PSU/PCIe-12V-connector for higher-power GPU (gaming or otherwise). And upgrading a rack of 20x 2U servers only adds 1400W, which you probably wouldn't even need to upgrade PDUs or add cooling to support.
This might even fit in my 4-node server.
why should I get this of the A380 low profile cards which is a fraction of the cost?
Try the Sparkle Eco as it is a low cost around $100 USD, OHH and its low profile single slot. For me I was wanting AV1 and a line in sound card for a used Dell Optiplex 7060.
swapped out my 1060 6gb for an RTX A2000 in my Unraid server a while back and its been working perfectly and i can do at least 6 transcodes at a time no issues, im sure it could even handle more. it was a steal at $270 after tax.
for a 12GB A2000?
I wish I had such a card for my SFF PC, without paying the workstation premium.
Does it fit in Minisforum MS-01? Edit: Nevermind, lost me when you said it costs as much as RTX4090. :(
Cant you use 2-3 rtx 4000 to reach the calculation power of the 4090? I dont know if its possible to do the inteference on multi gpu
Good question 🤔
Use Nvidia Container runtime. This helps you spin up Ollama inside a container. This you can now pass to a reverse proxy and use a hostname
I really do prefer this type of cooler over the dinky little milled/extruded axial coolers generally on low profile geforce gpus
Would Chimera OS also work? Or, not yet.
I'm hoping to get the A310 ECO for HW transcoding here in EU.
I am building a Z790-A I-9 with 2 fan cpu radiator and a 1500 psu, 2tb m.2...and 64GB RAM (possible 128GB in a month~s/h). I couldn't imagine needing a $1800-2200 GPU. If I wanted to make an ornate wood enclosure PC for someone, I would recommend. Not exactly something you can get replaced on the fly, I am already two-three weeks out on shipping as is, I would want onboard graphics in that case.
Perhaps it would be the equivalent of replacing a Tesla S glass roof vs the windshield. Start buying insurance, right?
still waiting for someone to use this with non-ECC memory, non-pro microcode and big passive heatsink at a reasonable price
keep the PCB though, it's important for the airflow
also, it's a beast, it's like 4070 kinda with 70W, come on, it doesn't get any better no matter how much you pay extra
I'm REALLY interested in ROCm and AMD's W7500 RDNA3 based workstation GPU for my home server!
The perfect replacement for my P4.
I don't plan on using any nvidia hardware and their propietary drivers, so I just hope this brings in a bit more competition on the SFF space, I love small and tiny gpus, as I don't play any AAA games and love SFF PCs.
using an A2000 for main gfx in an unraid setup Windows VM. IT's no 4090, but it does MED/HIGH?ULTRA settings in most games on 1080 steady down to 30FPS, barley any stutters. Solid little card. Looks like time for an upgrade.
First thing I did was check the price. Not gonna be home labbing with this anytime soon!
Could you do a review on the Nvidia Jetson.
Especially for the Jetson Agx orin!
Maybe a stupid question, but can this GPU fit into a 1U server?
I would be great to make it even passive cooled.
hows it handle gaming and streaming of games?