Having taken an interest in BWT yesterday, and working out my own implementation in Pascal, though not understanding all of the theory...this really clarifies a lot of it for me... thanks so much for having this out there for me to watch during this time of Stuck at Homeness.
26:15 why the answer is not row 800 instead of 801? If we are looking for G100, isn't it that we should skip the first 99 rows starting with G instead of 100 rows?
Great video. I see how BWT relates to compression, but also how it's a form of encoding. Does anyone know of a comprehensive list of reversible functions that can applied to strings or sequences?
Hi Ben at 20:30 You explain that Last column is sorted by right-context that is why we see same order as in F. But aren't middle columns sorted by right-context? Why for middle columns don't we see the same ranking order as we see for First and Last?
I figured that you've already figure out the answer by yourself now but I would like to add in a few lines for the newly comer later on :-) Ben gave excellent (i.e., short) explanation of the idea of the "right"-context sorted directly linked to these F-L chars list occurs in the same order in the original text. The reason is that, (1) by definition, First column reflects the sorted order of BWM, so, by eliminating the first char (these being the same char), the remaining string is remain sorted (that is, the right-context). (2) Eliminating Last column chars you get the original string (i.e. the right circular shift); then again, they are listed int the sorted order by definition of BW matrix. Thus we conclude that the F column and the L column reflects the same sorted order by definition of BW matrix! Now, for your question, these properties DO NOT hold in general. For example in the second column, the orders of a-char are in stead, the orders in first column are as shown at 16:55.
BWT brings like characters together in runs for example: if we pass the string "A man has got to do what a man has got to do" to BWT algorithm we get "TOOSSNNAATTOO.MMHHH......W..AATTDDGGAAAOO..." as the output. Yes it is the same length as the original string but you will notice that BWT brought some like characters next to each other like some the character A or O or H. Yes not all the character A are next to each other but it's better than nothing. Now exaggerating this result of BWT, say there was a 100 A next to each other and a 50 G next to each other, now if you pass this - somewhat - ordered output of BWT to a compression algorithm, it will notice the 100 A next to each other, so it will - for example - output like this: A( 100), and so for any other characters that is repeated. ls it clear now?
I had trouble with this initially as well. The answer is that B-ranking is performed according to the order of each letter's occurrence in the last (L) column. Then, since we know that F and L have the same rank order for a given letter, it follows that the B-rankings in F are also ordered from top to bottom.
Excellent presentation. I have one question. The $ is put at the end of the unique sequence. Can it be put at the beginning...I understand that the implementation would have to be changed to account for the positioning of the $. I suppose I am asking is there a particular advantage at putting the $ at the end, rather than another place. For instance, to sort 5' to 3' (assuming the left end is the 5' end) rather than going backwards.
Since the $ is just a marker, you can probably put it anywhere (so long as you consistently handle it) You can even remove it entirely, unfortunately when you do that you need to store an offset of which character then becomes your 'last character' or the offset to make the original sequence without rotations. (the offset data could take up more space than 8 bits though)
If you don't have an ending character with $, it isn't needed. However you will still need to store which offset which correctly decodes to the original string (without rotations). In Bzip2 it might be a 4 byte int, in a sting where $ is never used, it's 8 bits.
your code in 13:38 can simply in this way: def bwtViaSa(t): bw = [] for si in suffixArray(t): bw.append(t[si - 1]) . # list[-1] means the last element in list return ' '.join(bw) thanks :)
Love the video, but I feel as though you skimmed over the lexicographic sort very quickly without much explanation. @1:27 would be great if you had an intermediate step to explain the sort. Maybe understanding a lexicographic sort is a prerequisite to learning this, but if you are trying to start from nothing, it would be great to visualize that sort process! Otherwise amazing video, and love the reversal explanation! So interesting that many other videos do not explain the reverse given that the transformation is almost useless without understanding the reversal!
i cant watch this, it goes on... and on... and on.... and on... and on.... and on.... and on.... and on... ad infinitum. you take 37 minutes to describe something that could have been described in TWO minutes without ANY loss of information. you dont need to repeat the same thing 8 or more times in a row in different ways in order to teach!
Great video!
0:24 BWT
2:50 BWT implementation
9:34 Suffix arrays
12:48 Suffix array implementation
15:12 T-ranking
18:13 LF mapping
21:53 B-ranking
26:44 Reversing
32:24 Reversing implementation
Watched a dozen video/articles, finally understand why this algo works now. This video is the best. It explains the reversing part so well!
You got me when you made that 'blubb' at the beginning in order to show how the a is moved to the end hahaha
Having taken an interest in BWT yesterday, and working out my own implementation in Pascal, though not understanding all of the theory...this really clarifies a lot of it for me... thanks so much for having this out there for me to watch during this time of Stuck at Homeness.
I believe that this the first video I have seen on youtube with no dislikes ... very well deserved.
26:15 why the answer is not row 800 instead of 801? If we are looking for G100, isn't it that we should skip the first 99 rows starting with G instead of 100 rows?
The dollar sign adds an additional skip
This video was pure gold!
thank for sharing ben, you are a good teacher!
Thanks Ben! Very clear and I now understood everything on the BWT
Just found this video. Very informative and well done.
Great video. I see how BWT relates to compression, but also how it's a form of encoding. Does anyone know of a comprehensive list of reversible functions that can applied to strings or sequences?
Thank you Ben, this is invaluable!
Hi Ben at 20:30 You explain that Last column is sorted by right-context that is why we see same order as in F. But aren't middle columns sorted by right-context? Why for middle columns don't we see the same ranking order as we see for First and Last?
I figured that you've already figure out the answer by yourself now but I would like to add in a few lines for the newly comer later on :-) Ben gave excellent (i.e., short) explanation of the idea of the "right"-context sorted directly linked to these F-L chars list occurs in the same order in the original text. The reason is that, (1) by definition, First column reflects the sorted order of BWM, so, by eliminating the first char (these being the same char), the remaining string is remain sorted (that is, the right-context). (2) Eliminating Last column chars you get the original string (i.e. the right circular shift); then again, they are listed int the sorted order by definition of BW matrix. Thus we conclude that the F column and the L column reflects the same sorted order by definition of BW matrix!
Now, for your question, these properties DO NOT hold in general. For example in the second column, the orders of a-char are in stead, the orders in first column are as shown at 16:55.
Brilliant instruction, sir, thanks a lot!!
Great videos, thank you so much ... one question ... "We have already talked about suffix indexes" refers to which video?
Correct me if wrong. But since it's 0 based, shouldn't G100 be the 101th row of G in F column so in total it's the 802th row in F?
It's really helpful for me. Thank you so much.
BWT(T) is the same length of the orignial string T. How does BWT makes it more compressible? What infomation did you actually compress?
BWT brings like characters together in runs for example: if we pass the string "A man has got to do what a man has got to do" to BWT algorithm we get "TOOSSNNAATTOO.MMHHH......W..AATTDDGGAAAOO..." as the output. Yes it is the same length as the original string but you will notice that BWT brought some like characters next to each other like some the character A or O or H. Yes not all the character A are next to each other but it's better than nothing. Now exaggerating this result of BWT, say there was a 100 A next to each other and a 50 G next to each other, now if you pass this - somewhat - ordered output of BWT to a compression algorithm, it will notice the 100 A next to each other, so it will - for example - output like this: A( 100), and so for any other characters that is repeated. ls it clear now?
@@bahaaalaagmail Yep. After thought about it for a while. I came up the exact explanation as you do. You just confirmed my guess. Thanks a lot.
How to do the B ranking? I could not understand...
sort according to alphabet. first $ (assuming smaller than any char, or smaller than A); then all the As, then All the Bs, etc.
I had trouble with this initially as well. The answer is that B-ranking is performed according to the order of each letter's occurrence in the last (L) column. Then, since we know that F and L have the same rank order for a given letter, it follows that the B-rankings in F are also ordered from top to bottom.
Thanks, Ben. Nice explanation.
Thanks! It was really clear and helpful to me.
Brilliant explanation! Thank you very much!
Excellent presentation, thank you very much
Can you describe how the Burrows-Wheeler-Scott transform (aka Bijective BWT) works?
The second video of his my engineering prof sourced us 😂
Good explanation of easy decoding! (Vs sort, prepend, N times)
why do take the order of a's as a3,a1,a2,a0?
Excellent presentation. I have one question. The $ is put at the end of the unique sequence. Can it be put at the beginning...I understand that the implementation would have to be changed to account for the positioning of the $. I suppose I am asking is there a particular advantage at putting the $ at the end, rather than another place. For instance, to sort 5' to 3' (assuming the left end is the 5' end) rather than going backwards.
Since the $ is just a marker, you can probably put it anywhere (so long as you consistently handle it)
You can even remove it entirely, unfortunately when you do that you need to store an offset of which character then becomes your 'last character' or the offset to make the original sequence without rotations. (the offset data could take up more space than 8 bits though)
Thank you for this awesome lecture. I assume this BWT is used for exact matching.
Is there a video for inexact matching as well?
What is the need to append $ sign at the end of the string while implementing B-WT?
If you don't have an ending character with $, it isn't needed. However you will still need to store which offset which correctly decodes to the original string (without rotations). In Bzip2 it might be a 4 byte int, in a sting where $ is never used, it's 8 bits.
i dint understood how to sort the sequences?
rotate the input and then sort it
so you'd get an array of strings like:
abaaba$
baaba$a
aaba$ab
aba$aba
ba$abaa
a$abaab
$abaaba
Afterwards sort it.
Why to explaine the T-ranking?
to show they are in order from the left side and ride side, showing you can decode it using an easier method... or that's what i get from it.
code? please!!
This is such an epic algorithm
explained really well
your code in 13:38 can simply in this way:
def bwtViaSa(t):
bw = []
for si in suffixArray(t):
bw.append(t[si - 1]) . # list[-1] means the last element in list
return ' '.join(bw)
thanks :)
Goat
This has problems handling empty spaces, otherwise, it is pretty much perfect.
thanks, well-explained.
Thank you!
BWM???
Thank you
BEST SOOOOOOOOOOOOO FAAAAAAARRRRRRRR
Love the video, but I feel as though you skimmed over the lexicographic sort very quickly without much explanation. @1:27 would be great if you had an intermediate step to explain the sort. Maybe understanding a lexicographic sort is a prerequisite to learning this, but if you are trying to start from nothing, it would be great to visualize that sort process! Otherwise amazing video, and love the reversal explanation! So interesting that many other videos do not explain the reverse given that the transformation is almost useless without understanding the reversal!
Every lecturer should have youtuber skill for better explaining skill
Very useful, I coded a Java implementation off this presentation ( github.com/sorokod/Burrows-Wheeler-transform ).
i cant watch this, it goes on... and on... and on.... and on... and on.... and on.... and on.... and on... ad infinitum.
you take 37 minutes to describe something that could have been described in TWO minutes without ANY loss of information. you dont need to repeat the same thing 8 or more times in a row in different ways in order to teach!