Demo: Rapid prototyping with Gemma and Llama.cpp

Learn how to run Gemma locally on your laptop using Llama.cpp and quantized models.
Checkout more videos of Gemma Developer Day 2024 → goo.gle/440EAIV
Subscribe to Google for Developers → goo.gle/develo...
#Gemma #GemmaDeveloperDay
Event: Gemma Developer Day 2024

[@ayoubachak2154]

I've used gemma for a benchmark in a research project I'm working on, where I compared human results against AI, gemma was the closest after bloom 176B, followed by models like mistral instruct 7Band llama 34B, even the 2b version did pretty well, great work team 👏🏻

[@polish4932]

Hi mate, if you'd like to compare diff models for the same question, you do so on Wordware. Highly recommending it! ;)

[@ayoubachak2154]

@@polish4932 thank you

[@banzai316]

Very cool, thank you!
I like this format with demos. We are developers!

[@flynnmc9748]

This is a fantastic format for a talk, insightful and engaging for a viewer!!!

[@GoogleDevelopers]

Glad you enjoyed this video! 😎

[@user-eh7uo8hw2v]

0:21 🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉

[@judevector]

Wow this is so cool 😎, developers changing the world

[@arpitkumar4525]

Minimum System Requirements for running a model locally?

[@kevinkawchak]

Thank you for the discussion.

[@Daniel-zl7wf]

At 9:03, Gemma shows some solid satire

[@svenkoesling]

Just my two cents: 1. No explanation on how to connect LM Studio to the Llama.cpp, 2. newest hardware required - at least it doesn't work on my M1 with eight performance cores and 32 GB Ram

[@arpitkumar4525]

Really cool and simple to understand

[@ser1ification]

Thanks for the demo!

[@thesimplicitylifestyle]

I was looking for this! Thanks! 😎🤖

[@forrestfeng1098]

Like it very good sharing.

[@voidan]

how do you connect the LM Studio to llama.cpp? you used a preset which was probably custom.

[@parisneto]

CODE would be awesome, as well as knowing the SPEC of the notebook as it’s easy to buy a sub1k or 5K+ at apple store depending on so many factors…

[@zencephalon]

Good demo, nice tooling suggestions out of this

[@johnkost2514]

Wrapped in the llamafile runtime it is an even better single file .. oh yes!

[@MyEthan1998]

If anyone faces an error on Mac about "network error: self signed certificate", close the app and use terminal, run "NODE_TLS_REJECT_UNAUTHORIZED=0 open -a "LM Studio" "
This reopens the app and the error should go away. I have no idea where to put this info sooooo...

[@ChrisTrotter-oj9du]

good, thank you

[@cho7official55]

Cool demo, I'll try it

[@monamibob]

Very interesting demo!
What kind of extra work would be required to run this without LM Studio? Does Llama.cpp contain the necessarry functions to load models as servers you can interrogate?

[@some1rational]

Has anyone else tried doing this? I tried following this exactly with LM Studio using the exact model and prompt but I am consistently getting atrocious outputs; the gemma model is just outputting gibberish or incorrectly formatted JSON.
I wish there were more details on the presets used.

[@TheOrator_Ese]

Very nice 👌 cool Demo

[@MacGuffin1]

Great choice of demo app!!

[@amatuerIdeas]

damnn its using 15 gb of ram i have an 8gb m1 i dont think it will work for me..

[@lorenzo9196]

You can download a quantized version 8 maybe 4-5 bits

[@amatuerIdeas]

@@lorenzo9196 okay i will try

[@andreawijayakusuma6008]

did gemma should use GPU ? so I wanna try to learn this model, but I didn't want to use GPU

[@digisignD]

Cool. Will definitely use this soon

[@airhead2741]

Is this meant to be super accessible? If I have an APU, on a laptop with no GPU or NPU(?), that means I can expect it to run fairly well? Also considerations for a lighter yet usable model?

[@erickcarrasco1938]

I tried that in an old APU, very slow generations but the same result.

[@user-vq8on7dh1y]

Nah, Gemma is just a parot. It is released for fine-tuning, aka research purpose.

[@tonydevelopingstuff]

Very nice!!!!

[@bonadio60]

Very nice, but what is your computer spec? Memory and chip?

[@darthvader4899]

Probably m3 max 128gb

[@JJN631]

Gemma 7b can run on a rtx 4060 laptop

[@A032798]

How about windows environment? Is LMstudio/Ollama a better choice?

[@nayzawminnaing2562]

That's a lot of RAM to run this for me.

[@dtmdota6181]

Anyone notice ram usage of 16.68 GB? What was that?

[@takudzwamakusha5941]

This is so cool.

[@emmanuelokorafor1705]

It's cool now, but what if each application starts deploying local models. It'll turn our phones into what data centers were meant for thereby reducing costs for large corporations. Trading a few megabytes for faster and more expensive chips.

[@cmoncmona959]

Please Elaborate. What were data centres meant for? Asides hardware to run inference of worldwide requests. If it’s done locally, surely it’s better for redundant tasks. Also, the data centres use a lot of megabytes and expensive chips.

[@indylawi5021]

Very cool demo 👍. Any chance we can get the source code 😀

[@zoomatic293]

This is so cool :)

[@devagarwal3250]

Pls provide source code also

[@awakenwithoutcoffee]

where can we learn to set this up ?

[@KuldeepSingh-in6js]

cool demo

[@yubrshen]

What’s the required hardware specs?

[@deeplearningpartnership]

Awesome.

[@tandaramandaraba]

wow

[@AIPeter-dd9hr]

game using lm studio, interesting.

[@savire.ergheiz]

Just focus on your existing products Google. Which are a mess 😂

[@Killputin777]

never trust google products.

[@f00kwhiteblackracismwarsh07]

Google seems to be trying out too many new things. to me thats a turn off and red flag. everyone is different 🙂