GPU vs CPU: Running Small Language Models with Ollama & C#

In this video, we'll explore the performance differences when running Large Language Models (LLMs) in Ollama using both the CPU and GPU. Watch as I demonstrate a live sample in C# using Microsoft.Extensions.AI to run Ollama inside a Docker container. Curious to see how these models perform locally? Let's dive in and compare the results!
Useful links:
.NET Video Analyzer repository: aka.ms/netaivi...
Ollama in Docker: ollama.com/blo...
.NET & AI Show: • .NET AI Community Stan...

[@bdanuw]

Another great video from ElBruno :)
Thank you Bruno.

[@elbruno]

Glad you liked it!

[@eugene5096]

Thank you Bruno, interesting as usual !!!

[@elbruno]

@@eugene5096 Thanks! The CPU vs GPU is a wow one 😁

[@RomuloMagalhaesAutoTOPO]

👍

[@muluhsenet7582]

❤❤❤

[@bilalbilal7674]

Bilal here😊, I think you should go for creating an extension then it will be good and easy to accessible

[@elbruno]

There is one in the Aspire Community Toolkit: github.com/CommunityToolkit/Aspire/tree/main
I may record a video about that one!
Best

[@samirou976]

Hi ! This is very interesting but I wonder how it would perform on an NPU ? Is it possible to make it run on NPU ?

[@cuachristine]

NPU is a GPU with all the graphics bits removed.

[@samirou976]

@@cuachristine Yes I know thank you but that was not my question

[@elbruno]

Ohh that's a great question! I still don't have access to an NPU machine, however, if docker desktop allows access to the NPU, it should work. Let me ask Justin (a fellow CA.NET), who rocks the docker world to see what he can share about this.

[@samirou976]

@@elbruno Thanks man I would realy apreciate that 🙂

[@jimmymac601]

Does this support multiple GPUs ?

[@elbruno]

I'm not sure, I'll say no, we may need to check with the ollama team