Unicode, in friendly terms: ASCII, UTF-8, code points, character encodings, and more
Вставка
- Опубліковано 30 тра 2024
- Ever been bit by a Unicode bug? Maybe you weren't treating UTF-8 encoded data correctly, or tried to read it as ASCII? Maybe you mixed up UTF-8 vs UTF-16? Unicode and character encoding might seem like a tricky topic, but let's break them down and learn about them piece by piece, from ASCII to code points to graphemes to combining character modifiers and more.
Thumbs up emoji link: www.fileformat.info/info/unic...
00:00 Intro
00:12 All Data Is Stored As Bits
00:41 How To Store Characters Using ASCII
01:38 What About All The Other Languages?
02:14 Graphemes Map To One Or More Code Points
03:18 Code Points Map To One Or More Bytes
03:44 UTF-32 Is Wasteful
04:43 UTF-8 Is Better
05:40 Unicode is Western-Language-Centric
06:41 Mid-Video Recap
07:52 Live Python Demo
09:47 Unicode Rules Of Thumb
10:14 Looking At A Code Point
this alone is the best unicode video explanation in the entire youtube, 100x better than the, maybe, second place from Computerphile.
Computerphile's explanation was very concise as well. I don't understand why you are blatantly throwing shade on them like that lol
@@energy-tunes Computerphile presentation appears cluttered to me. While Alex seems to be tidy & to the point with flow reminiscent to that of a river.
@@ankitchabarwal6814 nothing about it is "cluttered". the difference is that computerphile covers the most important aspect of utf 8: how it knows how many bytes to read for multibyte codepoints, while this one leaves it out. i am not blaming this video, as it is a little confusing to explain, but this video is by no means the best.
@@rz2374 It is best in one aspect: Keep it to the general encoding itself. Only one enhancement in aspect to UTF8: Pointing out to look for the UTF8-encoding strategy elsewhere…
i didnt understand clearly ... Can you explain me?
Another fun fact about the way letters are laid out in ASCII: A capital letter's corresponding lowercase counterpart is exactly 32 values ahead. This is because 32 is a power of 2, and makes it so that you only need to flip the 6th bit (from the right) of a byte to change the case of a letter.
ASCII digits are also nicely laid out; the low nibble of characters '0' to '9' are 0 to 9.
Yup. You only need to XOR to flip back and forth
@@Michael75579 yup. You can subtract from '0' to get decimal equivalents.
This is how most assembly programers do it
NOT ONLY ASCII
I was going to say that it lost it's use in Unicode because it doesn't work with other alphabets, but went to check and… It works for Cyrillic and Greek as well!
1101000010010000 is А (cyr)
1101000010110000 is а
1100111010010001 is Α (greek)
1100111010110001 is α
1101000010010110 is Ж
1101000010110110 is ж
Edit: Although it doesn't work for every character,
1101000010101111 is Я and
1101000110001111 is я
@@bororobo3805That's not true, most programmers if they're good, will bitwise AND or OR rather than doing an arithmetic operation. With a barrel shifter, which most CPUs have, it will always be more efficient than using an arithmetic operation.
7:50 Worth pointing out that Python 2 is way past its end of life now and in Python 3, all strings are Unicode aware and the u modifier does nothing (it's only there for backwards compatibility). Using Python 2 is a good way to showcase the difference between Unicode aware and unaware functions but seems like this could confuse some beginners trying to replicate what you're doing who will likely be using Python 3 and might not be aware of the difference.
This also caught me off guard. Python 2 has been deprecated for such a long time by now.
Only officially and even then was another point release considered. For some platforms there is no py3 port and plenty of py2 code is still around being used.
@@mokovec It's definitely still out there but by now the official end of life has already been almost 3 years ago and it's finally dying for real. The consideration of "another point release" has long passed. On my distro, python 2 isn't installed by default anymore and on Windows, you need to really go out of your way to install it. Platforms where 2 is still the only option are far and few in between and certainly not the ones beginners are using. And there's definitely no reason to encourage starting to use it now.
@@1vader I don't see anyone encouraging it.
@@1vader this video is not a programming course, it's an explainer on utf-8.
i assume he chose python2 on purpose specifically to demonstrate the differences in a simple language (so that a beginner wouldn't get too distracted by the syntax) that has an explicit delineation between unicode and non-unicode strings. of course we shouldn't be writing production ready code based off this video lol, and the video isn't encouraging it (he literally says "check out the applicable string behaviour and libs in your own language"); it's just a teaching tool
FINALLY someone is able to explain this clearly. Most other videos complicate this so much. Thank you!
8:15 In python3, at least now it can do that with python without any problem.
>>> a="你好"
>>> a[0]
'你'
In python3, every string is in utf8. Using a u"" is a thing in py2
Technically, the internal representation of strings in Python is actually UTF-32 or UCS-4 (they're basically the same), because strings need to be efficiently indexable - as these encoding are uniform-length, that's easy to do. Using utf-8, the lookup of a given character in a string would take longer, because the whole string up to that point would need to be traversed as bytes are not evenly allocated to each character.
To top things off, strings that only contain Latin characters just use ascii anyway internally to reduce waste. That's why when a string is appended to, it creates an entire new string object, as the characters the new string has to encode may be outside the original encoding's range.
Great explanation, thank you! For anyone who may still not understand, UTF-8 CAN get up to 32 bits and bit as big as UTF-32, but only if it has to. Otherwise, it just uses the minimum amount of space (8 bits) and expands as required, depending on the grapheme.
you deserve a fucking award man, I swear, this is the top 1 explanation on UA-cam. NONE of the others are even close to yours.
Traditionally major Chinese coding methods would just consider a Chinese grapheme as 2 characters though, because they would use 2 byte coding while the the 128 ASCII code points only use 1 byte coding. And also, traditionally a Chinese grapheme would always take up double the width of an ASCII grapheme in fixed width console font. This kinda makes everything neatly aligned (the amount of storage bytes needed is the same as the amount of character printing space needed), but basically falls apart when Internet and UTF-8 become more popular.
And that's also basically the very reason there are double width Latin letters in Unicode. Traditionally it's used to improve readability of English words in vertical text arrangement for Chinese and Japanese, and is called full-width letters full width as in it's the full width of a Chinese character.
Where have you been all my life?! 🤩 What a great explanation! Thank you so very much! 🥰
I've listened to quite a few Unicode tutorials. This one blows the others out of the water. Clear. Concise. Good tempo. Thx
This is the type of content that should be suggested by UA-cam to everyone. Great explanation, thankyou
You have an amazing talent for teaching m8! Keep it up, this video like your others is so helpful. I love that you cover basic concepts of coding, not "how do I implement a server in node.js" but what really is the essence of becoming a good coder.
I've watched a few videos today trying to understand these concepts and this is by far the best. Good job
Man this channel is golden. The most difficult topics here are explained so easily and in such a unique way
This video is like the sum of the most important things about unicode and ascii, very well done.
👍👍🏾
this is such an appealing video due to the detail, music, subtle animations, even the colour theme. thanks for the video!
at 1:49 the arabic is rendered incorrectly. it shows each letter in the isolated form instead of the connected forms.
and it's written from left to right instead of right to left
happens every time lol, i remember some big project did the same thing recently
I took a long searching to understand the Unicode (also UTF-8 and the others), I did, but some things were still ambiguous to me, this guy literally taught me the whole topic with only 10 minutes
This is the best explanation of unicode and UTF-8. Super comprehensive & helpful! Thanks!
This video packed so much information in such a short amount of time, amazing. Thank you for this content and please keep uploading,
Absolutely love this explanation. As someone else mentioned, this is even better than Tom Scott's explanation on the numberphille channel.
This is such a great explainer video. As somebody who's run into the codepoint slicing problem in Python a lot. This explains so much! Thank You again!
Definitely the best explanation of Unicode I could ask for. Checked with Elixir and it appears that Elixir stringս are Unicode aware by default! Amazing!
wow your Chinese pronuciation is pretty good there @2:24
Terrific video dude, definitely deserves far more views
This is so awesome yet somehow has so few views.
These kind of videos are amazing for people self-studying CS. I hope you have lots more videos like this to make it easier.
Keep up the great work 🤘
Glad you liked it! Any topic suggestions for future videos?
The atmosphere in the video is so chill, and the video is informative, UA-cam recommend this to more people.
Best explanation with a fantastic presentation. Thank you so much
Thanks, a ton!!
Here is what I learnt from the video (I have added a few things that I knew earlier):
- UTF-8 (Unicode Transformation Format, 8-bit) is an encoding scheme for representing Unicode characters.
- In UTF-8, ASCII characters are represented using a single byte, which means that any valid ASCII text is also valid UTF-8 text.
- Therefore, UTF8 is backward compatible with ASCII.
- In UTF-8, characters that can be represented using a single byte (i.e., ASCII characters) are represented as themselves.
- Characters that require more than one byte are encoded using a combination of multiple bytes.
- A code point refers to a numerical value assigned to each character or symbol in the Unicode standard.
- Code points are represented using hexadecimal notation and are typically prefixed with "U+" to distinguish them from other numerical values.
- For example, character "é" (Latin Small Letter E with Acute) consists of two Unicode code points: the base character "e" (U+0065) and the combining acute accent (U+0301). When encoded in UTF-8, "é" is represented by the bytes 0xC3 0xA9.
- A grapheme refers to a visual unit of a written language. It represents a single user-perceived character or a combination of characters that are displayed together.
- len() function returns the number of bytes, not the number of characters in a Unicode-unaware string.
- len() function returns the number of characters in case of a Unicode-aware string.
thanks for posting the summary here
The notes on pythons functions are wrong as they are for a much earlier version that has not been in use for half a decade
I have waited for this type of video for years, thank goodness you made one!
Excellent video! Explained everything in a way that I could understand very well, thank you :)
I kinda hoped this video would teach me how to use all the unicode characters in 11min, but it turns out it's just a general overview of how UTF-8 works. This is like going to a math class and learning that addition exists, but not learning how to use the + man.
loved this video! So so clear ✨
Easy to digest explanation! Good job, thanks 👍
Trying to be better in IT and dreaming to be an awesome programmer! I have always just skipped on learning Unicode and never cared due to laziness. But now realize the very importance! thank you so much for this video.
Very useful and applicable explanation of ASCII, UTF and UNICODE
The reason I'm going to watch this one is that it explains new stuff like Emojis without expecting already having knowledge about Unicode. Bad informative material usually either lacks the new stuff or expected to much knowledge about the old.
Such best videos better than 100X bad books, I can't stress too much how good this video is. There are always treasure videos like this on UA-cam, hope OP keeps working, keep letting your knowledge and legacy to influence more people.
Very good explanation. Clearly deserves more views.
Hey thanks for the explanation; I spent some time understanding the Unicode with the tables ASCII table but this explanation is really good. Thanks.
I hit the (U+1F44D) button because this video was so easy to understand!
honestly thought you had a million subs. quality is top notch.
been using Unicode for so long but never really understood what it means. Thanks for explaination.
Thanks for showing actual code.. it made it very easy to understand
Great video. Thanks for explaining this clearly. This is the best video on Unicode
this is the best video I've seen for how unicode works! Thanks!
This is the best explaination of Unicode.👏🏼👏🏼👏🏼
Your tutorial videos are amazing. I decided to go back to creating soft after 16 years. soft soft is so easy to get into, but also offers
Excellent Explanation Man... Understood every bit of it... Many Many Thanks😊❤️
Very good explanation , thank you for your work!
Quality content, awesome. Post more these kind of great explainable video
So much informative video, thank you for creating such a video, and also salute to your effort for making this video. 👍 ❤
Beautifully Explained
wow, best explanation i've seen... short and accurate. Bravo.
thank you so much alex , great and lucid explanation .
This was very informative and clear. Thank you.♥
Damn, that was such an amazing video!!! Learned so much from this, wish my lecturers would've explained it like that as well ahah
Thanks, this video really helped me understand text encoding better.
I'm a programmer so this knowledge is really useful :)
cleared all my doubts. best explanation on the entire universe :)
Absolutely amazing video. Good Job 👍.
Thank you so much, this helped a lot!! You're a great teacher, your explanation was amazing!!
Such clean video format for explanning clearly a knowledge ! Thank you^^ (by a newcomer in the field of informaticL
May you have a nice day with cookies and tea/milk 🥛🍪
no videos were explaining clearly the step of converting the "grapheme" to a "code point" and then to binary they were just assuming that middle step was understood. that's what I was missing apparently thanks great video
Brilliant explanation Alex, thank you
Beautiful man, you're the only one who helped me
This is so nicely explained
Excellent, thank you - helpful and interesting!
Amazing explanation about Unicode
WHY? WHY? WHY HAVENT I SEEN YOUR CHANNEL, YOU ARE THE BEST HANDS DOWN, definitely sub + bell (and like)
just the way you explain.. its so good and clear
Yeah, I don`t know what is going on with my channel. Is UA-cam ripping me off? I know the Freedom Fighters [world-wide] are.
The clearest Unicode has ever been in my career!
Thank you.. this was easy to understand with your explanation
Amazing job! thank you again!
i really apreciate your help with dowloanding this software
Hey! Thanks so much for this video!
great content, it's much more clear now. Thxxxxxxx
Awesome explanation 👍
Great video , made me understand finally.
subbed because of this wonderful explanation
thanks
TNice tutorials one is really good, among all other basics videos
Thank you so much for this!
Thanks man!! You've earned my respect
Your videos have helped reach over $200,000 in stocks by age 23! Thanks In The setup. Keep the videos coming.
Great video, thank you!
SO MUCH HELP WITH THIS VIDEO
Amazing video!
Hi! This is Gold! Please do more videos like this. especially the terminal colors
Thanks a lot! Best explanation ever
Awesome video, thanks a lot!
Thank you so much for the video, it really helps a lot!
ok i finally understand why i need to specify utf 8 when writing to files
I've got a paper that tells exactly how you can convert code point to UTF-8 bytes and vice versa. It's actually pretty good piece of knowledge and it tells exactly how UTF-8 encodes code points. Had I known that before, I could simply convert the Unicode symbols by hand myself.
Great video, the menu works great
great explanation! it would be a good idea to explain how encoding would prevent in injection attacks.
yes it works brother ! many thanks
Fun fact: most writing systems can express their characters in 2 o 3 UTF-8 bytes (only 2 for the most used ones: extended latin, arabic, cyrilic, greek…). The only outlier is CJK Ideographs (Chinese, Japanese and Korean). There are an insane amount of them so each CJK grapheme carries more meaning (like a word) so it makes sense that they take more bytes in UTF-8 (usually 4 bytes).
Thank you Alex!
Best tutorial on Unicode
Awesome video
Thank you !
Thank you so much.