Hugging Face Tokenizers (11.3)

This video discusses how to use HuggingFace tokenizer models. These models break text down into tokens, which might be a single word, or make up multiple words. Tokenizers are the first step in natural language processing.
Code for This Video:
github.com/jeffheaton/app_dee...
~~~~~~~~~~~~~~~ COURSE MATERIAL ~~~~~~~~~~~~~~~
📖 Textbook - Coming soon
😸🐙 GitHub - github.com/jeffheaton/app_dee...
▶️ Play List - • 2023 PyTorch Version A...
🏫 WUSTL Course Site - sites.wustl.edu/jeffheaton/t8...
~~~~~~~~~~~~~~~ CONNECT ~~~~~~~~~~~~~~~
🖥️ Website: www.heatonresearch.com/
🐦 Twitter - / jeffheaton
😸🐙 GitHub - github.com/jeffheaton
📸 Instagram - / jeffheatondotcom
🦾 Discord: / discord
▶️ Subscribe: ua-cam.com/users/heatonresea...
~~~~~~~~~~~~~~ SUPPORT ME 🙏~~~~~~~~~~~~~~
🅿 Patreon - / jeffheaton
🙏 Other Ways to Support (some free) - www.heatonresearch.com/suppor...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#PyTorch #nlp #huggingface

[@mickelodiansurname9578]

thanks for that Jeff!

[@PicaPauDiablo1]

Always great. Thanks Dr Heaton

[@fernandoguerrero2895]

Hello Dr Heaton. I am planning to go back to school in 2024 for an online master in AI. I am planning to buy a new computer. I usually go for laptops but I am not sure if a PC gaming laptop or a Mac laptop would be enough for AI . Maybe going for a desktop would be better. But also I have seen in some forums that users don't even bother with buying a dedicated AI workstation and they use cloud. What would be your recommendation? Thanks in advance