Timestamps: 00:00 - AI terminology and technology 03:54 - Chips, semiconductors, servers, and compute 05:07 - CPUs vs GPUs 06:16 - Future architecture and performance 07:12 -The hardware ecosystem 09:20 - Software optimizations 11:45 -What do we expect for the future? 14:25 - Sneak peek into the series
In the usual case of floating-point numbers being represented at 32-bit, is this why quantization for LLM models can be so much smaller at around 4-bit for ExLlama and making it so much easier to fit models inside the lower amounts of VRAM that consumer GPUs have? Incredible video, interviewer ask really though provoking and relevant questions while the interviewee is extremely knowledgeable as well. It's broken down so well too! Also, extremely grateful to a16z for supporting the The Bloke's work in LLM quantization! High quality quantization and simplified instructions makes LLMs so much easier to use for the average joe. Thanks for creating this video.
It's a trade-off between accuracy and space/performance (i.e. being able to fit the model on local hardware). A 1-bit number could represent (0, 1) or (0, 0.5) as it only has 2 values. With 2 bits you can store 4 values, so you could represent (0, 1, 2, 3), signed values (-2, -1, 0, 1), float between 0 and 1 (0, 0.25, 0.50, 0.75), etc. depending on the representation. The more bits you have the better the range (minimum, maximum) of values you can store, and the precision (gap or distance) between each value. Ideally you want enough bits to keep the weights of the model as close to their trained values so you don't significantly alter the behaviour of the network. Generally a quantization of 6-8 offers comparable accuracy (perplexity score) with the original, and below that you get an exponential degredation in accuracy, with below 4-bits being far worse.
Guido Appenzeller is speaking my language. the lithography of chips are shrinking while consuming lots of power. Parallel computing is definitely going to be widely adopted going forward. Risc-V might replace x86 architecture.
"To remain competitive, large companies must integrate AI into their supply chain management, optimizing logistics, reducing costs, and minimizing waste."
A slightly different way of looking at Moore's Law is not about being "dead", but rather becoming irrelevant. Quantum computing operates very differently than binary digital computation, it's irrelevant to compare these two separate domains in terms of "how many transistors" can fit into a 2D region of space, or a FOPS performance. Aside from extreme parallelism available in QC, the next stage from "here" is in optical computing, utilizing photons instead of electrons as the computational mechanism. Also, scalable analog computing ICs (for AI engines) are being developed (IBM for example) . . . Moore's Law isn't relevant in any of these.
AI and cloud computing face power supply issue as cryptocurrencies? "Cryptocurrency mining, mostly for Bitcoin, draws up to 2,600 megawatts from the regional power grid-about the same as the city of Austin."
![The moment we stopped understanding AI [AlexNet]](/img/n.gif)
For a sneak peek into part 2 and 3, they're already live on our podcast feed! Animated explainers coming soon.
a16z.simplecast.com/
doesn't look like part 2/3 are up on the podcast feed (anymore at least) - any chance those video explainers are coming out still?
The music is very distracting. Please tone down in the future
Timestamps:
00:00 - AI terminology and technology
03:54 - Chips, semiconductors, servers, and compute
05:07 - CPUs vs GPUs
06:16 - Future architecture and performance
07:12 -The hardware ecosystem
09:20 - Software optimizations
11:45 -What do we expect for the future?
14:25 - Sneak peek into the series
In the usual case of floating-point numbers being represented at 32-bit, is this why quantization for LLM models can be so much smaller at around 4-bit for ExLlama and making it so much easier to fit models inside the lower amounts of VRAM that consumer GPUs have?
Incredible video, interviewer ask really though provoking and relevant questions while the interviewee is extremely knowledgeable as well. It's broken down so well too!
Also, extremely grateful to a16z for supporting the The Bloke's work in LLM quantization! High quality quantization and simplified instructions makes LLMs so much easier to use for the average joe.
Thanks for creating this video.
It's a trade-off between accuracy and space/performance (i.e. being able to fit the model on local hardware). A 1-bit number could represent (0, 1) or (0, 0.5) as it only has 2 values. With 2 bits you can store 4 values, so you could represent (0, 1, 2, 3), signed values (-2, -1, 0, 1), float between 0 and 1 (0, 0.25, 0.50, 0.75), etc. depending on the representation. The more bits you have the better the range (minimum, maximum) of values you can store, and the precision (gap or distance) between each value.
Ideally you want enough bits to keep the weights of the model as close to their trained values so you don't significantly alter the behaviour of the network. Generally a quantization of 6-8 offers comparable accuracy (perplexity score) with the original, and below that you get an exponential degredation in accuracy, with below 4-bits being far worse.
Well done, very clean and clear. Love your simplicity
Guido Appenzeller is speaking my language. the lithography of chips are shrinking while consuming lots of power. Parallel computing is definitely going to be widely adopted going forward. Risc-V might replace x86 architecture.
An excellent primer for beginners in the field.
This is highly informative and easy to understand. As an idiot, I really appreciate that a lot.
Great video. Tip of the computation innovation
"To remain competitive, large companies must integrate AI into their supply chain management, optimizing logistics, reducing costs, and minimizing waste."
Love this Channel! Could we also look at the hunger for energy consumption and the impact for climate change?
No wonder nvidia don't care about consumer GPU anymore.
Yup, cash grab
This is a good video.
Excellent video. Thank you and well done
Really helpful thank you!
Incredibly useful!! Thanks.
1:24 Ehm… I would like to know, what camera and lens/focal length you use to match the boom arm and background bokeh so perfectly 🤐
I use the Sony a7iv camera with a Sony FE 35mm F1.4 lens! I should note that good lighting and painting the background dark does wonders though too
Thanks for video but 4 mins before getting to any details in a 15 min video?
This was very good
This video makes clear WHY DSP [digital signal processing] chips were implementing sum{a[i]*b[i]} in hardware!
Older Vox style animations FTW!
See you at NY Tech Week
A slightly different way of looking at Moore's Law is not about being "dead", but rather becoming irrelevant. Quantum computing operates very differently than binary digital computation, it's irrelevant to compare these two separate domains in terms of "how many transistors" can fit into a 2D region of space, or a FOPS performance. Aside from extreme parallelism available in QC, the next stage from "here" is in optical computing, utilizing photons instead of electrons as the computational mechanism. Also, scalable analog computing ICs (for AI engines) are being developed (IBM for example) . . . Moore's Law isn't relevant in any of these.
Great job
Good one, Thx.!
The Render network token solves this
it would be so cool if this main speaker was a clone
Huang's law
AI and cloud computing face power supply issue as cryptocurrencies?
"Cryptocurrency mining, mostly for Bitcoin, draws up to 2,600 megawatts
from the regional power grid-about the same as the city of Austin."
Back to School Giveaway
Geforce 256 aka GeForce 1 wasn't even Nvidia's first gpu let alone the first ever PC gpu... 😅😂
The future
AI power consumption has doomed it to failure before it has started?
ua-cam.com/video/lRy5Sy9Elbw/v-deo.html
The music is unnecessary and actually annoying.