From Words to Tokens: The Byte-Pair Encoding Algorithm
Вставка
- Опубліковано 11 тра 2024
- Why do we keep talking about "tokens" in LLMs instead of words? It happens to be much more efficient to break the words into sub-words (tokens) for model performance!
very well explained.
Subscriber from substack