Advantages of Letter based Tokenization for Machine Learning

Let's chat about letter-based tokenization in machine learning models. TikTok folks asked about the advantages of using letters for tokenization, especially when dealing with the attention mechanism. Well, there are several.
Letter-based tokenization gives us fine granularity. It catches more details since it's looking at each character, which is crucial for rare words and nuanced meanings. It also handles misspelled words better.
Instead of needing a huge dictionary of words, we keep the vocabulary small, just letters, numbers, and some punctuation, making it less compute-heavy and more flexible. Plus, it keeps the input size consistent, which helps when working with varying word lengths. With character-level cues, models generalize better, even across different but similar words, adding some noise robustness like managing typos and slang.
This method works great with various models like RNNs, LSTMs, CNNs, and transformers. Now, speaking of models, you might wonder if big ones like GPT-3 use letter-based tokenization. They mix it with other methods, but early models like the char-CNN or deep emoji heavily relied on it.
Newer ones like CharFormer are also exploring it. Other tokenization methods include word-based, sub-word, sentence-based, and n-gram tokenization. Each has its perks and downfalls, but a mix often works best, especially in large-scale models.