With the automatic audio dubbing from UA-cam /Google you hear a synthetic voice in your regional language. To hear my original voice in English, switch to "Default" or "English" in the settings. Thank you.
I wonder if using a 2000 word vocabulary but instead of simple language use a specialized domain vocabulary for domain specific expert Tiny LLM might work. I.e. is it the language simplicity used or the vocabulary size that affects the result?
Tiny Language Models offer fascinating potential! They could transform AI in Edge Devices and multi-agent systems. Checking out the UIUC research. What practical challenges do we anticipate?
I've been wondering about progressively building up a language model by training these small models to be extremely strong for their parameter count, and then embedding them inside the layers of an LLM while freezing all of the small model layers except the first and last (to allow it to adapt to the larger model as it is trained). Would be interesting to see the results.
I wonder if you could apply the Meta's LCM technique to tokenize concepts rather the text and further improve the performance of the model. In a sense, I guess you're transferring some of the burden from the LLM to the tokenizer by doing this in that, the concept tokenizer itself would have to be trained. It might turn out to be more efficient for these smaller networks to work on wrangling concepts rather than wasting precious resources inside the tiny LLMs on converting tokens into concepts.
With the automatic audio dubbing from UA-cam /Google you hear a synthetic voice in your regional language.
To hear my original voice in English, switch to "Default" or "English" in the settings. Thank you.
I wonder if using a 2000 word vocabulary but instead of simple language use a specialized domain vocabulary for domain specific expert Tiny LLM might work. I.e. is it the language simplicity used or the vocabulary size that affects the result?
I'm SOOOO excited for this. Thanks for making me aware.
Always loved your video, your video always give me spark of new ideas & project i want to work on. Ty sir
Tiny Language Models offer fascinating potential! They could transform AI in Edge Devices and multi-agent systems. Checking out the UIUC research. What practical challenges do we anticipate?
I've been wondering about progressively building up a language model by training these small models to be extremely strong for their parameter count, and then embedding them inside the layers of an LLM while freezing all of the small model layers except the first and last (to allow it to adapt to the larger model as it is trained).
Would be interesting to see the results.
This is a very helpful concept!
I wonder if you could apply the Meta's LCM technique to tokenize concepts rather the text and further improve the performance of the model. In a sense, I guess you're transferring some of the burden from the LLM to the tokenizer by doing this in that, the concept tokenizer itself would have to be trained. It might turn out to be more efficient for these smaller networks to work on wrangling concepts rather than wasting precious resources inside the tiny LLMs on converting tokens into concepts.