- 20
- 13 109
Build With Binh
Приєднався 15 жов 2020
Binh is a HCI researcher at MindRove, previously worked at Ericsson as an Innovation Engineer and Intel as a guest developer.
Building real-time conversational AI on ESP32 (using LiveKit and WebRTC)
🧑💻 Project Description
This is one of the very few projects that demonstrate real-time conversation AI on ESP32. Furthermore, unlike other projects, it is the first to display the use of LiveKit, with audio output and input. This is used to resemble Wheatley, from the famous video game, Portal 2.
💻 Github Repository
The source and written tutorial of the project can be found here: github.com/pham-tuan-binh/wheatley-ai
🛒 SenseCap Watcher
If you would like a SenseCap Watcher yourself, consider buying it from my affiliated link: www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html?sensecap_affiliate=3gToNR2&referring_service=link
This helps me a lot since I'm not yet in UA-cam monetization program and making these videos cost me quite lot as a creator.
Seeed Studio Coupon (applicable to most items on their shop): 5EB420ZS
👨💼 Collaboration
The project is distributed under a copyleft license, details on Github. If you want to collaborate with me or commercialize this project, please write me an email at binhpham@binhph.am.
#esp32 #embedded #ai #livekit #elevenlabs
🎞️ Chapters
00:00 - Beginning
00:54 - Chapter 1
01:46 - Chapter 2
02:06 - Front end
03:21 - Back end
04:44 - Chapter 3
09:35 - Demo
This is one of the very few projects that demonstrate real-time conversation AI on ESP32. Furthermore, unlike other projects, it is the first to display the use of LiveKit, with audio output and input. This is used to resemble Wheatley, from the famous video game, Portal 2.
💻 Github Repository
The source and written tutorial of the project can be found here: github.com/pham-tuan-binh/wheatley-ai
🛒 SenseCap Watcher
If you would like a SenseCap Watcher yourself, consider buying it from my affiliated link: www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html?sensecap_affiliate=3gToNR2&referring_service=link
This helps me a lot since I'm not yet in UA-cam monetization program and making these videos cost me quite lot as a creator.
Seeed Studio Coupon (applicable to most items on their shop): 5EB420ZS
👨💼 Collaboration
The project is distributed under a copyleft license, details on Github. If you want to collaborate with me or commercialize this project, please write me an email at binhpham@binhph.am.
#esp32 #embedded #ai #livekit #elevenlabs
🎞️ Chapters
00:00 - Beginning
00:54 - Chapter 1
01:46 - Chapter 2
02:06 - Front end
03:21 - Back end
04:44 - Chapter 3
09:35 - Demo
Переглядів: 1 486
Відео
I made an AI teddy bear that can talk and feel
Переглядів 2,6 тис.14 днів тому
👀 Making an AI teddy that can talk and feel with “sensory inputs” it runs completely local and in real time ⭐️ About GLaDOS This project couldn't be done without GLaDOS Personality Core, a project for making real life GLaDOS from the Portal series: github.com/dnhkng/GlaDOS 🌟 Source code for this teddy bear You can find the source code of this project here: github.com/pham-tuan-binh/GlaDOS-but-T...
I deployed AI agents on Mesh Radio (they can call emergency service for you)
Переглядів 232Місяць тому
👉 About this demo The demo is to present a novel use case of ToolCall deployed on a local LLM node on a Meshtastic network. The agent is implemented and deployed on my open-source platform. The source code can be found here: github.com/pham-tuan-binh/radio-llm/ The agent can be used to automatically call emergency service for user in distress, detailing and summarize info for dispatchers, and g...
I made the first off-grid LLM Platform on Mesh Radio (Meshtastic)
Переглядів 4,3 тис.Місяць тому
🌟 The world’s first LLM platform which is accessible through plain radio stack using Meshtastic. 💻 Open-source: github.com/pham-tuan-binh/radio-llm 🙌 What differentiates this and LLM which has been on Meshtastic before: - Whole range of LLM features such as message retention, knowledge, user separation,… - Support bot interaction like Discord in addition to LLM chat. - Support tool call for age...
I made a 5G camera management platform and pitch it at Nokia HQ in Finland
Переглядів 180Місяць тому
This project was hacked in under 24 hours by me and my friend, Luke from Aalto University. It was for 5G NaC showcase at Unbreakable Connectivity hackathon by Nokia and Telia. You can find the full demo in my UA-cam Channel. Multicam footage example was a snippet of @JustinOdisho’s video on editing multicam footage ua-cam.com/video/MQIj1sZEck8/v-deo.htmlsi=ac679xpiyLTiNwCg #nokia #telia #hackat...
This is what I pitched at Nokia HQ in Finland
Переглядів 115Місяць тому
This is a demo for ReSync software suite, made by me with NextJS Recoil Nokia Network-as-code SDK. Together it pairs with a TimeSync protocol made by my friend and the footage daemon, which was simulated in the video as we couldn't make it in time at the end of the hackathon.
I made an AI that predicts my weight
Переглядів 1552 місяці тому
If you want to have a look at the code, here is the google colab for all the code in this project: shorturl.at/nydLZ If you have any idea, comment below. This is still a very early stage project and I hope to build its accuracy and use cases even more in the future. Have a gooddd time watching!
I went to Trieste and Ljubljana (and made a short film about it)
Переглядів 742 місяці тому
I went to Trieste and Ljubljana (and made a short film about it)
We celebrated mid-autumn festival in Budapest, Hungary
Переглядів 1364 місяці тому
We celebrated mid-autumn festival in Budapest, Hungary
I spent a day at Hortobágy National Park in Hungary
Переглядів 1174 місяці тому
I spent a day at Hortobágy National Park in Hungary
How to make a fkn good video, for amateurs
Переглядів 2084 місяці тому
How to make a fkn good video, for amateurs
nếu có thể mong anh làm video chi tiết về các bước thực hiện nó
Các bước thực hiện chi tiết a có để trên github trong description đó e, nếu có gì vướng e có thể nhắn a.
Anh sinh ra và lớn lên ở Us luôn ạ?
@@Ellen_le567 không e ơi, a ở Hải Phòng hết cấp 3 rồi a qua Hungary học
@@build_with_binh Anh học đh nào ở Hungary vậy ạ?
@@Ellen_le567 a học BME (TU Budapest) nhé e
Anh hiện đang lập trình nó bằng VS code trên ubuntu đúng ko anh, liệu e dùng arduino ide để lập trình thì có gì hạn chế hơn ko anh 🤔
@@phucle1885 có nhé e, về bản chất thì a đang dùng esp-idf để làm firmware thay vì arduino. Arduino cho esp-32 mà mng hay dùng thực chất là wrapper của esp-idf nên khá limited về các tính năng sâu hơn giả sử như quản lí task, quản lí core,… Trong project có một số component sử dụng đa luồng của eép32: thread gửi audio được gắn vào core nhanh của esp32, còn thread nhận audio gắn vào core chậm. Điều này giúp e chạy 2 task gửi nhận cùng lúc.
@build_with_binh em cảm ơn lời giải thích của anh ạ. Em thấy rất hứng thú với project này nên muốn thử làm rồi cải tiến lên một xíu nên hy vọng sẽ được anh hỗ trợ giải đáp thắc mắc trong quá trình làm sản phẩm ạ.
cho nó nói giao tiếp bằng tiếng việt được không anh nhỉ
@@KissgamEvn được nhé e, a đã test với tiếng đức, tbn và tiếng việt
@ hay quá a
Wonderfull. Maybe im stupid, or just oblivious, but do you still use the realtime API from Open Ai, or would it work with an openai assistant API with help of an TTS and SST? ?
@@antiprenor it’s okay, i didn’t use the realtime api from openai, i made it work with a custom pipeline, Whisper -> GPT4o -> ElevenLabs
Respect
Appreciate it man 🤜🤛
wow
glad u like it 🫶
Anh có dùng thử gemini multimodal live api chưa? Call realtime bằng websocket, ngon hơn nhiều, mà chỉ hỗ trợ voice eng output
6:31 kind enough :)))
Finding that repo was insane haha, was 1 step away from writing my own livekit signaling implementation and they just had a hidden repo lying somewhere
Great video! If you can figure out to do the same thing with some more power you'll achieve so many nerds wet dream.
Thank you, that dream is definitely a milestone haha
very nice video, wondering what type of camera do u use to film your stuff. Thanks!
I use my phone (Iphone 15PRM) to film the video. To ensure the high quality, I film in Apple Log and color grade it manually afterwards.
Làm hết tất cả là cần bao tiền vậy anh
@@dangne911 cái sensecap giá 2 triệu nma về cơ bản thì e có thể chạy firmware trên bất kì board audio esp32s3 nào, giả sử như sonatino.com nếu e tự chế thì giá project có thể dưới 500k.
@build_with_binh dạ cảm ơn a ạ
Amazing<33
Tks 🫶
Bro, you are wild. Thank you for being an inspiration
My pleasure, thank you for the kind words.
I see no views so i gonna support you one, you deserve it!
Tks man! Really appreciate it 🙌
Michael Reeves type shit
Great Job! just change the voice its too scary ,need to have a more soft "tedy" tone. keep us updated
Thank you! Will do!
hay quá !
This is so awesome!
Thank you!
This is so cool. I wonder is there a way to improve the tone of the bear? For now it sounds so much like a robot
@@lamvu-q1z nothing that we couldn’t do with local models or hosted models, but there are tradeoffs regarding latency. If you do local models, the high quality models are going to sound extremely good but takes some time to generate. If you do online models, the round trip time is large already.
oách xà lách vô cùng
That's way too cool! Suscribed to your channel :)
Awesome! Thank you!
nice. so funny
This is sooo cooool!
Tks man!!
i want one
Let me waitlist you haha, priority line
Nice job my fellow Viet builder 👏
ngoài lề bác có thể chia sẻ cách học tiêgns anh ko ạ
Về ngữ pháp hay vốn từ thì mình học từ các trung tâm dạy tiếng anh từ bé (lớp 6) đi theo giáo trình cambridge, r lên cấp 3 mình đi học các trung tâm dạy ielts để luyện thi. Cái này mình nghĩ bây giờ có thể tự học miễn phí được, nếu cố gắng. Còn về phát âm, nói chuyện và đọc thì mình sử dụng và nghe tiếng anh hàng ngày, cũng từ tầm lớp 6 do mình hay vọc trên các forum nước ngoài và xem youtube. Mình nghĩ chỉ cần ứng dụng và tập kĩ năng nhiều là được, để nói được như này mình mất tầm 6-7 năm.
Good job bro 🥰
Tks bro 🫶
Amazing
Tks man 🫶
Fan TASTIC
Awesome video, good work! It is cool to see people finding new ways to use this, however it is "Mesh-tastic" lol Need more height and better antennas to increase that distance!
Thank you for a great meeting dude!
Thank you as well, you did wonders for the first time on tape dude!
HOLY THIS IS INSANEEEEEEEEEEE
@@goodgamist5113 glad u like it!
So cool! Well done! 🎉
@@insanity54 tks man, glad u like it 🫶
wow thats fast! super cool
Đỉnh quá bro
@@paxle cảm ơn bro!
You should ad the open ai tool. Process the message with OpenAI api.
@@Flowglowlow support for Gemini and ChatGPT was actually in the first version, but as i developed it further and use more platform features like tools, it’s harder to maintain and debug multiple providers at once so i just went with local for now, but i will look into other options soon, perhaps something that let you configure between different llms, both offline and online
Applications / Services over meshtastic is a cool idea. My college's amateur radio club has been helping our sommunity build a mesh over the last year and I'll have to show them this.
@@sethproaps8899 That’s great to hear, you’ll find even more new features in the github as well, i think it will be perfect for your use case The platform now supports functionalities like a discord bot i.e calling /command to do stuff + allows llms to use tools by themselves.
Subbed, deserves more views! Would love to see a reticulum implementation of this!
@@HotMicSystems thanks man! Glad u like it, I just know about reticulum today, will look into it, sounds fun
Brother needs to lay off the adderal, THATS INSANITY!
@@doktorphlog2143 😂 I wish, glad u like it
Very cool idea! I have to try this myself some day.
Great work. Btw coverage of meshtastic network is highly dependent on antennas and reasonably free fresnell zone between them. If you have clear line of sight between antennas, both on the hills or high building, range can be tens of kilometres. When I took my meshtastic to hike in the mountains, I received messages from nodes 40km away when I was on the top at altitude 1200m
@@skopyhoTechChannel glad u like it! Definitely, I’ll try the setup with clear line of sight next (from a mountain top in Budapest), hoping I can reach the same distance as you
Really cool experiment!
@@levisorenson7873 ty glad u like it!
Great work!
@@afk1448 tks man, glad u like it!
Thats great creative technology mahn...very innovative
@@otisrancko ty glad u like it!
thấy trên facebook, bèn sub luôn ạ
@@vl30.7 cảm ơn bác 😁
Congrats ❤🎉. You beat me to the punch with this project. Much success with this great new project as it 🚀🚀
@@BetterThanTV888 ty! Hope I didn’t stop you from making your own, would be great to see your implementation as well
it’ll be amazing all the things our digital agent will do on our behalf-
Very interesting idea Binh, keep it up. Subbed!
@@tungdinh4114 thank you anh!
This is really cool! I would love to try this myself but it is the first time I hear about meshtastic, which lilygo devices are you using? Thanks and great video
@@Lordcomets tks bro! I’m using their lora v2.1-1.6, it’s pretty bare-bone and probably the cheapest but it works
@@build_with_binh well I got 2 t3s3 and tried it, your code works very well!