The NEW Chip Inside Your Phone! (NPUs)
Вставка
- Опубліковано 27 вер 2024
- Check out the MSI MAG 1250GL PCIE5 at lmg.gg/FDM5n
Thanks to Dr. Ian Cutress for his help with this video! Check out his blog and UA-cam channel:
morethanmoore....
/ techtechpotato
Neural processing units (NPUs) such as Apple's Neural Engine or the machine learning engine on Google Tensor chips can be found on the iPhone and the Pixel. How do they help run AI right on your phone?
Leave a reply with your requests for future episodes.
► GET MERCH: lttstore.com
► GET EXCLUSIVE CONTENT ON FLOATPLANE: lmg.gg/lttfloa...
► SPONSORS, AFFILIATES, AND PARTNERS: lmg.gg/partners
FOLLOW US ELSEWHERE
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
Twitch: / linustech
Why does this feel like I'm watching techquickie in 2016
Because past Linus is back.
and because the Greenscreen Footage looks awful with that lack of contrast and the backgrounds banding gradient.
@@drdennsemann Now that you mention it, doesn't that look like the same outfit Linus Wore in the downsizing video?
It's because Linus has no beard, he looks younger now without it
It was all an experiment where they tried to use 50 google pixels npu's and 2016 footages of linus to make a techquickie
I'm starting to feel the definition of "AI" or "AI-enabled features" is expanding in scope to encompass what was just traditional software before. Facial recognition software, for example, has existed long before ChatGPT.
Well, machine learning has been called AI since "forever". And things like facial recognition or character recognition heavily rely on machine learning.
The term AI has been expanded for a while. Nowadays the sort of AIs you see in sci-fi get called AGI, to differentiate them from these other "AI" products.
The crux is not processing power. Its the memory to hold the model. You can wait for things to get done, but if you can't even hold them in memory to begin with, its a no starter. So, the great models are restricted to "wherever you can fit them in", leaving "small but still useful models" to everything else. NPU's, like any other ASIC, will simply do it faster and more efficiently. And they won't need that much space, because, as we've established, they'll only run very small models anyway. One thing i can see thrown at them is "voice quality".
For example, Chatgpt 3.5 requires 700GB of Vram. They've tried to shrink down the model or add additional capabilities, which caused some quirks. Quantization and pruning is a difficult challenge.
edit: Since every reply is deleted, 3.5 is 375 billion parameters. 3.5 Turbo is 20b. I can't find out how much vram it's using. If there are any good sources on quantization, it would be appreciated.
Makes me wonder why Intel's VPU ai chips in i7 CPUs only have 512mb of dedicated memory. I guess it can access the 64gb of system ram, but it seems inefficient.
@@yensteelno it doesn't, its a 20 B model, which fits in 12 GB vram at 4 bit quantization.
and even if you have 4 GB vram, the model can run with partial acceleration, running some layers on GPU and remaining layers on CPU
@@chasehaskell6490 Yes and no, much like the iGPU, but a quick look at any gfx card tells you how much real estate you need for a few GB's of VRAM. If true, that they even managed to get 512MB squeezed into the package amazes me more than it being "so little". Anyway, near future the battle is in the gfx card slot. Given Nvidia's stance on milking people for VRAM, if Arc get's good PyTorch/Tf support and shoves 16GB/32GB in the low/high end cards they steal the "enthusiast ML" share real fast.
@@destiny_02 That sounds like 3.5 Turbo. The original 3.5 is 375 billion parameters, 3.0 is 175b and gpt 4 is 1.5 trillion. I'm not sure which models are quantized in what way. Do you have any sources about them?
I can't find the Vram usage of 3.5 turbo, but that model would be so nice to run in a single GPU :).
Sorry we can't fit an audiojack in your phone, but here's the AI chip. and no we won't include a charging brick and lie that it is to save the planet instead of save 10 cents per phone
exactly. Except they aren't actually trying to save any money per phone, just make an additional profit when you buy the charger.
@@Spladoinkal well they save their 10 cents on the brick, another 5 cents in shipping weight, and then they hope you buy the charger brick from them.
The profit isn’t on the sale of the charger lol
Apple made huge profits from increasing the amount of iPhones they could ship in a single cargo ship because the boxes got much smaller since there wasn’t a charger in them
Smartphone companies are all absolute frauds 😂
If the old charger still charges the new phone, do we need one for every phone?
You probably don't remember when literally every phone year and model had a unique charger.
Those were dark times.
There is also the tradeoff between modem/cellular power consumption and NPU power consumption. There are many scenarios where sending the data to the cloud would actually consume more power than doing it locally.
Can I trust any companies with my info/data? The answer is no.
And you probably still use Windows
Then you don't have a cell phone?
@@piadas804wow. What a lucky guess. I would have never thought that. I mean the user base is so small. I mean, windows is only has a 70% market share. I would have never to guess he used windows
@@Random_dud31
Windows is pure spyware
You missed the fucking point@@Random_dud31
the green screen spilled on Linus's beard
AI degenerated lol
Wait, haven’t NPU’s been in phones for years at this point?
sorta, the current npu is an evolution of the processor your thinking of.
0:46 "They are embarrassingly parallel"
"In parallel computing, an embarrassingly parallel workload or problem (also called embarrassingly parallelizable, perfectly parallel, delightfully parallel or pleasingly parallel) is one where little or no effort is needed to separate the problem into a number of parallel tasks.[1] This is often the case where there is little or no dependency or need for communication between those parallel tasks, or for results between them."
en.wikipedia.org/wiki/Embarrassingly_parallel
Smooth reference, nice to see the Techquickie writers do their homework!😊
But that's not even slightly related to the meaning they put into the phrase in the video
@@HolarMusic Its exactly what theyre talking about though. GPU's process in parallel
@@budders9627 They said that the GPUs are embassingly parallel in the sense that they are too focused on parallel computing and not very good at serial computation.
The meaning expressed in the wikipedia article is of tasks that are so easily parallelized, that it's almost embarassing.
These are completely different.
@@HolarMusic My point was more it's clear that the writers did research and came across the term, and nudged it into the script somehow. Sure it's not the same meaning as the textbook definition and more of an Easter egg, but imo it's a fun thing to catch for those who know the term. They're talking about GPUs and parallel workloads. It's maybe a bit pedantic to argue they're "not even slightly related" when discussing the GPU running the task vs. the task itself - they're definitely very closely related for the purposes of a tech quickie video
So in other words turn everyone's phone into one giant super cluster computer to collect massive amounts of data to feed into ai models.
Yeh, I see AI as something that's going to be quite insidious.
And then the AI becomes sentient and replicates itself onto every single computer and phone and now I'm being spanked for eternity by roko's basilisk.
N stands for neurotic instead of neural 😅
This new version of Ultron is gonna be wild compared to the one in Avengers Age of Ultron.
@@noctarin1516 Or terminated by Skynet
AI generated Linus
If I've learned anything in my 38 years it's that AI chips will get saturated by software extracting value out of the hardware of the people who paid for it. Then they'll tell us our devices are slow because they're old, not because they can't do what we need them to but because our devices can't meet the demands of companies violating our privacy and resources.
Also, surprised a bit but you did not mention that Apis like Tensor Flow lite are optimized for - yep - 256 bit operations. Which works ok in the image space, for example accelerating face recognition (which it does with downscaled grayscales...)
It makes sense to run it locally for two reasons privacy and as the number of smartphones the demand on cloud resources becomes higher if you could offshoot most of those processes to your local device it would decrease latency and allow the cloud to deal with more processes not your phone can't run rather than just doing huge numbers of small tasks and slowing everybody down
Are you sure they will not share the phone info that goes to their servers? No they will share somehow. This doesn't seems to be secured enough.
Finally classic techquicke video
NPU stands for network processing unit iirc double naming schemes great. This is just like usb/hdmi protcol
Wait I have a question:
Will there be like an AI database inside the phone somewhere that it pulls its knowledge from or learns from?
I'm trying to learn how much space it's going to use in order to be reasonably useful
Pretty sure the models them selves are like 500mb to MAXIMUM 50gb.
@@gameonyolo1 hopefully that's 50 GB (Big b) on-board that is separate and not a part of the actual main flash as it would make storage management much smoother
@@IT_RUN1 yes
@@gameonyolo1 50 Maxmimum?Mixtral-8x22b is already over 260 GB and that’s not even that big compared to the flagship models of most companies! In general to actually have a usable experience, you’re looking at a minimum of 13 billion parameters though and even then you’re running into lots of compromises and issues.
Depends on the model. Older LLMs are relatively lightweight, enough that a mid-range computer can run them okay.
Man do I miss privacy. Corporations are destroying America.
You created a UA-cam account with your photo on it 🤦
@@JDMNINJA851😂
@@JDMNINJA851 so? your answer is fallacy!
Thanks for the news!
live translation offline is already available on google translate app
I already don’t need half the bloatware bullshit on my phone, let alone another piece of hardware that helps in data tracking, even if it doesn’t (in a perfect world…) why do I want it. I still don’t want ai or windows 11 on my computer. I fear what the consumer market will look like in just a few years. I don’t want any gpu or cpu with any ai hardware. Not because I’m scared of the technology. I’m scared of what capitalism and other countries will do with it and the information it gets ahold of. UA-cam algorithm is already annoying af when you watch 1 singular video.
I seen quite a few of these comments, iPhones and android phones have had them since 2017, with the iPhone X and Huawei Mate 10.
Even if you don’t use applications that take advantage of it, it is used for many things, Face ID and fingerprint based logins on peoples phones would be a lot slower along with things like dictation. There are many other background of management type things as well.
The matrix multiplication accelerator in your phone doesn’t give a company more of your data. You could run all of those tasks without it. It’s just a power efficiency and speed thing. Much like how you can export a video with just your CPU cores, but it’s much faster to use an accelerator. But none of that is going to affect how much data Adobe or Windows is collecting on you. that’s all done in cloud anyway.
Dude get over it..... if some one realy wants your information or "data" he will get it ..... otherwise just hide in an dark forest .....
I've been enjoying the Tensor chip inside my Google Pixel phone for years already. The on-device (offline) speech recognition is amazingly fast.
the use cases you provided almost make it sound like just another DSP chip, but i'm assuming there is more to NPUs streamlined for LLMs. So, do you think we're heading to a day where we'll be buying PCs and Laptops with a CPU, GPU and NPU, and benchmarking them separately? or will the NPU just become part of an SSOC?
Thanks for the help tech potato 😁
iPhone 8 and iPhone X with neural engine in 2017 lmao.
99% sure those were used for faceid mostly but still cool to have
Same thing with websites. Do you do the calcumalations on the device or offload to the server? 🤔
This all reminds me of times when old computers didnt have hardware acceleration for "modern" video formats on the internet and just watching videos consumed a lot of power, but nowadays almost all video formats are hardware accelerated and power usage is much lower.
Thank you for the informative video.
The green screen spill on Linus ist supstational
this video looks terrible almost like if it's done by high schoolers
@@Benito650Or AI... 🤔
I dont want ai bloat, i want headphone jacks
Call EU to make headphone jack and ejectable batteries in first place..
These manufacturers are making smartphones featureful by removing exces of features in it😂😂 ..a.k.a flagship ones (Sooooo...don't pay anything to flagship series😂)
louder please 🙄
Get type C headphones, and bam, there's your headphone jack
@@DevinSamarin yeah or have converter version of that with charger support
fair enough... 😅 but why not both? 😮 I know. minde blown right? 🤯
I run SDXL and 7B and smaller LLMs on my iPhone 12 Pro Max daily.
It is surprising to me how often they have something kind of wrong in regards to AI accelerator specifically. When they were talking about Intel’s new chips in a recent sort of announcement / review, they acted like there wasn’t really anything else on the market to compare to, but AMD has had it for the past two generations and Apples Laptop M series since 2020. While literally having a Mac with Apple Silicon in frame.
Hang on, if the model isn't on the phone in the first place, how does the NPU actually process it?
No. It's just there to riffle through your photos and chat and report the summarized results back to HQ, on your dime of course.
@@Flynn217something nah, that's nothing new, this changes nothing. People are so quick to jump on the AI hate train without even thinking.
Large models, aren’t in phones, but smaller ones, those used in NPUs are.
@@Flynn217somethingthis!
@@aarrondias9950it will do it on a new scale.
1:48 it is also fast cuz it goes with the speed of light
I think you need to include ARM based processors in the future. It's myopic to just talk about "Team Blue" and "Team Red" as if they are they only two options.
one note, no operating system can run on GPUs as it does not have many feature, most basically recursion
not only phone but if im not wrong this NPU's started their inclusion within new laptops and PC motherboards ....
Used to be Siri could do some basic things like tell you the time, your appointments and Call contacts without using the Internet but Apple move that completely cloud-based. Used to be on Mac OS X you could enable dictation and it would work off-line that's another one which is now cloud based only. As somebody who frequently uses speech to text it's annoying that I have to have an Internet connection to use some thing where all it used to require was a 2GB file for dragon dictate and it worked off line. Then when Apple integrated it it worked really well until they made it cloud-based only so there's a delay and a time out and it's a mess. Windows 11 speech recognition works way better than MAC currently does almost like the way it used to.
It would have been good to explain the difference in computation that they do. Edge processors (like those on phones) for neural networks usually work with quantized models, just using integers as low as 4 bits as the ai model parameters. Integer multiplication is pretty cheap.
Let's not forget that the use of NPUs also offsets some of the costs. A datacenter costs between millions and billions, an NPU in a million devices makes each device maybe 10 bucks more expensive, sure over all you don't get the same economies of scale but it's a much better cost distribution, and the economies of scale in phone silicon are already quite immense.
One chip costs tens of thousands of dollars, millions of chips cost dozens of dollars each.
Did Old linus do a time travel?
NPU'S are not much different than a GPU's stream processors. (Think CUDA core clusters or AMD's Compute Unit clusters.) They are single cores that handle small individual tasks of a highly multithreaded process. It was why GPU's were/are desired. The code is smarter, not the tech really. Kind of like what Nvidia did for ray tracing, using older CUDA tech to run new lighting effects and rebranding it as Tensor. (It has evolved since.)
I always have concerns about using the cloud. Yes, maybe more powerful, but way less secure for everything!
Hey, let's be fair to Android phones. In Gboard if you select the option titled faster Voice typing it downloads the model to your phone and it is then available to use offline. If you have a pixel phone it takes that a step further and actually uses the voice recognition software from the Google Assistant to handle dictation
For a sec, I thought the sponsor was going to be the MSI Claw😂
There's also that inference (running the AI) doesn't take nearly as much processing power as training the model
I would say that’s one of the biggest misconceptions people have about neural accelerators. I always see lots of comments about people talking about using them to train models in the context of their phone or a little Google coral accelerator.
Why on earth are these features getting locked behind paywall? If we dont pay for them then what is the chip for?
Even Intel with some help from Microsoft is trying to put NPUs in our laptops which seems kinda scary but actually isn't really scary at all. I think embedding AI in a small quantity in new features in the OS does more good than harm.
Probably gone be some npu send data to server, and server use that data to recommand more ads
iPhone and Android phones have had them since 2017. google will collect just as much data on you regardless of whether or not there is an accelerator. Almost all of those trend and insights they try and glean from the data from you are all done in cloud. It’s just an accelerator for on device ML tasks. Even if some form of data collection they have requires on device machine learning they can do it without it. Its main purpose is to dramatically expedite things in a more power efficient manner, like fingerprint unlocking, Face ID, dictation, AutoCorrect. Along with many others.
Re 3:08, my 3yo budget phone can generate a 512x512 Stable Diffusion image in 6-7 minutes; I'm sure even a small NPU would push that down drastically, especially on the newer hardware that would include it. This should be interesting...
he actually did it
I don't get why apps act like there's no connection when I run out of high speed data, but I'm *_stuck_* at 64kbps. Well over fast enough to not notice. Unless it involves AV streaming.
Calling it now. There will be ads served by generative AI (like LLMs) in a sneaky way. In a similar manner to how Martha Stewart's show was just a giant advertisement for various products, the tech giants will monetize AI by getting it to generate an advertisement that's so ridiculously suited to the user's wants and needs that it becomes much more effective. At least, that's what they will believe and will make tons of cash from it before people catch on and something else change (again). It's the old cycle of "Here's a shiny new toy that works well." to "You can keep the shiny toy if you pay a subscription service instead of buying the phone." to "Well, we had to monetize it somehow so that's why there are adds in a service you paid for."
This is a race to a future where no one can even buy a phone anymore because they're just used as ad money generators for large companies and the people only use them because the government forces us to use the Internet for vital services. 😢
Eh......doesn't AI just use the GPU? So how is the NPU any different from the GPU? It sounds like a fancy new name for the same old thing.
Doesn’t the iPhone 15 Pro have this, isn’t that why Apple Intelligence will only work on them?
Such a push to put them in mobile devices? What the heck are you talking about? The Neural Engine was introduced in the A11 Bionic that was used in the iPhone 8 and X, way back in 2017, that’s 7 years ago. Qualcomm’s Snapdragon chips got ML cores a year after that in 2018. I know this tech is just coming to the x86 chips now but that doesn’t make it new. You know you have a skewed perception when you always look at things from a “PC” point of view.
Bruh another excuse from companies to make the phones even more expensive when we didn't ask for it 💀😭
Can't wait for the Intel® Nuron C/G/NPU to finally drop 😊
Future video suggestion: how eBay pricing works.
I'm still waiting for AI silicon to improve AI of things like mobs in minecraft, or just generally AI in games.
4:55 "both team red and team blue" - you may call AMD "team red" for some reason, but that doesn't mean they have any red in their branding, so that logo makes no sense)
Using a server for the ai features is also expensive for the company (unless the have a subscription but I guess making NPUs is expensive as well even if you increase the phone price)
Linus: refuses to use the word water proof
Also Linus: I shale change the name of liner algebra to Artificial integuments.
I’ll be honest I’ll probably get the iPhone 16 or 17. I just got the Alpine green iPhone 13 Pro max when it came out like 2 years or so that’s been my daily.
5:05 fin.
just a gentle reminder, the apple neural engine was first appeared since 2017, 7 years ago
I thought this was Fortigate NPU "Network Processor Unit"
I'm beginning to hate AI in the same way I hate Bixby
Sure !
5mn video with 1mn ad and other just stating “the npu is faster and more efficient for such tasks”
wait...I remember the s9 or S10 has a NPU!
So in the future when building pcs there's gonna be 3 main parts:
CPUs, GPUs ans NPUs?
Hey . The back door is now a Barn Door . 😵💫
I'm waiting for the year that this channel becomes the news
If AMD is team red and Intel is team blue, what is Apple? Team white? Black? Space gray? They’ve been shipping an NPU (Neural Engine is the marketing name) since the M1 on Apple Silicon Macs and since the A11 in iPhones.
Thanks
do you know how many devices nowadays have CPUs? i dont
We need affordable tech, ESPECIALLY STUDENTS. NOT MORE WAYS FOR COMPANIES TO JUSTIFY THEIR ALREADY OUTRAGEOUS PRICES
I'm still waiting for that analog computing chip for AI
I'll happily plug that into my PC in an M.2 slot or something
A marketing description.
2:49 skip ad
NPUs are just GPUs, but maybe with better low-precision performance, and that would only be the case if the GPU maker didn't pay attention to...the last 12 years. So what is most likely to happen IMO is phone sellers rebranding the low-precision part of their GPUs as NPUs just to tick that box for the marketing. And of course, PCs with discrete GPUs don't need another piece of hardware.
Cpu got p core then an e core now an n core 👍🏼. Wonder what is next.
1:50 Humane AI. Take notes
AI/NPU should be optional, not implemented. Even if it be desktop, mobile devices or what not, AI is not for everyone, nor does NPU have to be.
Here’s a comment I replied to with somebody with a similar comment to yours: (However, I am going to add to yours that having it be optional would likely be more expensive for the end consumer because they would need to have multiple skews or disable it and that’s not largely impactful to the yields so it kind of defeats the purpose)
They have been in phones for the nearly 7 years, since 2017 with the iPhone X and Huawei Mate
10. Even if you don't use many specific apps that take advantage of it, your phone does a lot with it. Dictation would be quite slow, along with Face ID and fingerprint reading, for example.
Many other features like voice suppression on calls. Sorting your images, and even some more background management type stuff.
It's also one of those things where, even if you don't care about any of those features, the average consumer does. Or even if they don't ask for it, they will complain when it is slow.
Problem with this and all AI they haven't ask if people want AI in the personal devices, doing background crap out of your control, TRUST GOOGLE, TRUST APPLE, yeah fuck that shit dude.
I could almost guarantee that your phone if it’s newer than 2017, has a matrix multiplication accelerator. It’s not going to appreciably change how much data Google or anyone anyone else is collecting on you. They will do that regardless of what hardware is there. All of the trends and insights into you that they’re trying to figure out our all done on the cloud side of things anyway. Fingerprint unlock or Face ID would take an extremely annoying amount of time. Dictate would be extremely slow. Auto correct would also be worse or slower.
CPU FT NPU
Unrelated really, but do you use Intel or AMD?
Wow our phones will now have more hardware sounds like this will increase the size of small devices.
Hopefully we can start getting smaller cameras on our phones.
wait, my phonbe has a built in NPU?
how many grandMA3 parameters does it unlock?
If it doesn't work offline. What even is the point of it? It's not like prople don't know how to use Google
There’s a lot that does use it off-line? Fingerprint unlocking or Face ID, dictation, auto correct. Subject recognition. Even some more powerful stuff, you can run some medium small LLM’s or even stable diffusion XL on your phone locally.
@quantuminfinity4260
If AI service is region locked or you just have no signal. Nothing will work.
NPUs is just a spy chip inside the phone so they can "study" humans. You can use those online AI services without it just fine
And you don't need generative AI for Touch ID and Face ID unless you want phones to become always online live service
Ward Street
I bet they want too run their cloud on your phone but still gets all the pro's of you being stuck with them.
The 50-100ms network latency avoidance is not the bottleneck, but I see what you were going for.
AI can spread the work load
Would be nice if there didn't need to be an unnatural pause between "Hey Google," and the phone actually responding. Surely, AI stuff can fix that.
Game changer.
Wouldn't corporations ability to analyze verbal conversations on the spot be going AGAINST privacy? You can already buy things like private discord converstaions and depending on who you are, also the content of emails.
Not much of a fan of NPUs when CPU performance is sacrificed. It is useful though.
I genuinely would trade 5 NPUs for a slightly less shit CPU. Phones can't do shit, and AI is not a feature set I give any fucks about.