16:30 You can also use [A-z] to match any upper or lower case character, because uppercase comes before lowercase in the ASCII table. That syntax actually means "match any character in the ASCII table between these two characters".
"\+" is nothing to do with the shell - it's because grep uses Basic Regular Expressions which doesn't include "+" as a metacharacter. If you use egrep (which uses Extended Regular Expressions) you won't need the "\".
I use grep to cheat a text based game called 'hangman'. I use GNU grep "-w' option to word matching.. Example: $ grep -w 'v.ir.s' /path/to/dictionary/file Thank you.
The problem for me with regular expressions is the learning curve with using them efficiently, and since I only need to use them infrequently, I never get familiar enough with their use, to use them to their best advantage. If I were using them all the time, I would not have to keep starting from scratch, learning how to use them.
Great video! BTW “asdf./“ is actually a valid URL since all domain names technically end in a period after the TLD. Most software infers the period if it isn’t specified, which is usually the case. Specifying the period at the end is actually the more correct format. So for example “google.com./search” is valid and should work in any software that accepts a URL.
thank you for video, i recently have been trying out DWM with dwmblocks, and wanting to write scripts for the statusbar, this was very helpful. im finding shell scripting very interesting and fun. i sure want to learn more. hope there's a follow up advanced video.
Is there a reason you were using double quotes instead of single quotes? I would be more inclined to use single quotes with grep and egrep to avoid accidental expansion problems. Generally I only use doubles when I know that I really want expansion. Here is an example to demonstrate the difference. Add two lines to your rt file: The first is the sentence "Navigate to your $HOME directory." Then add the absolute address of your home directory to the file. You could do this with the command "echo $HOME >> rt" Now the results of grep "$HOME" rt and grep '$HOME' rt give very different results. Also, I tend to use either egrep or grep -E instead of just plain grep. This is in part because I cut my teeth on Regular Expressions in Perl, and egrep is closer to the Perl that I learned first. It is also considerably more powerful.
now that we've found all instances of a certain thing in a text file, can you make a video on deleting, moving, replacing, etc. -- putting to use the output we've gotten here? thanks a lot. chad-like teaching content as usual
Regular expressions is one of those things I need once in a while, wind up spending a bunch of time creating something that looks like I hit my head on the keyboard, then forget how I did it months later. It can be simple, but it can also start to get long and tiresome if the requirement is more complicated, like matching any valid IP address "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"
5:04 yeah it's from the regural expressions of formal languages (more specifically Regural languages which are equivalent of L3 languages, those that can be generated form a right-linear grammar) where x* means {x}* so basically (x^0, x^1, x^2, ... ) x^0 is epsilon or lambda (also known as the empty word, a word of 0 letters) and x^1=x, x^2=xx, x^3=xxx and so on
5:28 {x}^+ is basically the same as {x}^* but without the epsilon/lamda, so it's (x^1, x^2, x^3...) As you can see this have something to do with Math more specifically with monoids. More infos here: www.ncbi.nlm.nih.gov/pmc/articles/PMC3367686/ en.wikipedia.org/wiki/Regular_grammar
I think newbies out there would also like to know that and \t also have special meanings: end of line and tab respectively. I'm not sure if that works with grep but it does work with other tools like python. I use them all the time
Hey Luke, sorry if this is too personal, but I noticed that you've started to display a lisp. Did you recently get Invisalign braces (or has the social isolation lowered your pronunciation level)? I had braces put on in my mid 20s and they made me a little lispy too
@@FyahBurn95 It is like I said, 'Z' comes before 'a' therefore it is invalid. [A-z] works, but includes a few non-letters, like [, |, ] and @. Search online for ASCII table.
@@elandje Have you tried it? [A-z] does not work with grep for me, but [a-Z] does. I know what the ASCII table is and the fact that A-Z comes before a-z, and also that UTF-8 is an extension of ASCII, which is what matters unless you actually work with ASCII files.
Good introduction with good examples! To throw something useful into it, don't you find weird you need to escape '+' but not '*'? And '*' is a (famous) shell expansion, but '+' is not! Also, if this was the case, it would be solved by the usage of double quotes you do, which avoids expanding anything but the dollar sign (for variables), or by switching to single quotes, which don't allow for any shell processing. The actual thing happening is that grep uses default regular expressions and the plus sign is from extended regular expressions, which grep understands but only if they are escaped. To use extended regexps without the need for escaping their metacharacters, try egrep or grep -E.
Nice work. I like your videos on general command line tools. Can I suggest a presentation on sed or awk? I know these tools may require longer videos but I m sure you can manage it
How are you quickly saving the 'note' file? I know ZZ is similar to :wq, but what's the similar command to :w? Where would I find the docs to read about commands like ZZ?
boomer, zoomer, doomer, coomer. I can' t keep up anymore, I must be gbetting old. Apparently I am a boomer now according to the zoomers even though I never was a boomer before.
What I miss in this video: What to *actually* use regex for. Here's what I've used regex for: Refactoring code - I had some functions in JS that I wanted to turn into lambda's. Replacing HTML in several files - The files were partially identical and II wanted to replace something in the header elements. Find non-ascii letters - I copied Haskell code from a book and it didn't compile, because the book used unicode quotes for the comments and GHC broke on those quotes. Find the nth comma in a CSV file - I wanted to remove everything after the 3rd comma or something like that, because I didn't need that data - the file was too large to open in Excel. At the end of the day regex is a tool to serve a purpose. Don't learn it for the heck of it. Learn it because you can use it to solve problems.
@@der0keks Huh, didn't know the cut command was a thing! I'm more of a Windows guy, so I'm woefully behind on my terminal knowledge. Happy to hear someone has found something useful! The regex was [^\x00-\x7F] BTW, which is basically searching for any NOT ASCII char.
Will you be able to design a regular expression which matches exactly the words a^n.b^n (a random number of 'a' followed by the same number of 'b') ? ;-)
So the criteria is that it must match instances of ab where the number of a's and b's are equal? So ab or aabb would match but abb or aaaaabb would not match?
@@thefantasicm_2407 Hmmmm, I'm not a regex guru by any means but I feel like there would almost have to be something out of regex to do the analysis? I feel like trying to come up with a solution now.
@@magetaaaaaa This is a tricky question : the answer is that it is impossible to recognize this language with regular expressions, sorry :-). This is related to the Kleene theorem. Look for Pumping lemma (regular languages) to understand why.
@@thefantasicm_2407 Hmmmm, maybe something like this would do the trick with Python. import re file = open("textfile.txt", "r") for line in file: a = line.count('a') b = line.count('b') if re.match("^a+b+$", line): if a == b: print(line)
You match with regex but how you can extract ONLY the matches? For example from a text file like your I eant to match and print out ONLY the emails , how can you do it?
@@GooogleGoglee Actually it is quite compicated. There are many patterns to be matched .. Try one of the solution in this URL. Thank you. www.unix.com/shell-programming-and-scripting/181361-sed-regex-extract-email-address.html
* means actually anything anytimes, right? I mean, you don't need to write .* Also, you've said that * means any number of previous character, but, in fact, it doesn't have to be specifically the previous character, it can be any charactar. Can someone tell if I'm right or wrong? Because I have learned that regex a couple of years ago and, since then, I use them the way I describe.
In the shell, and I'm sure in some other places, * does mean any character but I think the standard Regex grep uses, the basic and the extended one as well as PCRE, treats it as _any number of_ so you would have to say .* to match any character
"Crucifixion, well that's a nice thing as well." - Luke Smith, 2020
Luke confirmed for Longinus. Why else would he be so fluent in biblical languages?
@@internetfriendsimulation9156 hes redpilled
I feel like Luke's head is always cut off in his webcam view because his humongous chad brain wouldn't fit even if he went full screen.
It's a chest cam
@@sch8836 that's hot
He does that for the same reason he keeps wearing the sunglasses over his head.
@@yoyojuninho6130 he feels naked without them
Vegapunk ???
2:06 grep stands for "global regular expression print". It is an ed command: g/re/p. Ed is the standard text editor.
was about to say this, +1
Ed is the standard text editor.
@@Jack-hd3ov same
ua-cam.com/video/NTfOnGZUZDk/v-deo.html that's just the backronym
Brian Kernighan himself said came from g/re/p
Finally, I needed a regex tutorial.
Sure u did
16:30 You can also use [A-z] to match any upper or lower case character, because uppercase comes before lowercase in the ASCII table. That syntax actually means "match any character in the ASCII table between these two characters".
"\+" is nothing to do with the shell - it's because grep uses Basic Regular Expressions which doesn't include "+" as a metacharacter. If you use egrep (which uses Extended Regular Expressions) you won't need the "\".
@@juxuanu egrep is a thing. search it.
@@juxuanu No, sorry, misunderstanding, i thought you said "as far as i know its grep -E, not egrep"
@@minhajsixbyte egrep is deprecated
@@sk8sbest oh. i know very little about these things actually.
but why is this depricated btw?
@@sk8sbest oh thanks I have searched and got the answer!
I use grep to cheat a text based game called 'hangman'.
I use GNU grep "-w' option to word matching..
Example:
$ grep -w 'v.ir.s' /path/to/dictionary/file
Thank you.
thanks for the information bro. الله يبارك فيك
...or for crosswords.
Your channel has taught me more useful knowledge than college did
You must have gone to a terrible college.
"I want a period" - Luke Smith, 2020
The problem for me with regular expressions is the learning curve with using them efficiently, and since I only need to use them infrequently, I never get familiar enough with their use, to use them to their best advantage. If I were using them all the time, I would not have to keep starting from scratch, learning how to use them.
I swear I haven't really understood regex until now. Thanks for sharing your knowledge
Words can’t describe how much I’m grateful for what you did
Luke could do a 10 video series on Regex and still not scratch the surface. I encourage this content !!!
Great video! BTW “asdf./“ is actually a valid URL since all domain names technically end in a period after the TLD. Most software infers the period if it isn’t specified, which is usually the case. Specifying the period at the end is actually the more correct format. So for example “google.com./search” is valid and should work in any software that accepts a URL.
15 years working as a developer, at last someone made regular expression easy! I finally understood! Thanks Luke!
To be honest this is one of the best videos on your channel Luke. Short and informative, thank you.
I really like the way you explain Linux stuff!!
thank you so much, this is so useful and in depth. teach you basically 90% there
I literally sit through the whole thing effortlessly. great video
super super, gracias por compartirnos este video. muy ilustrativo.
I was recently thinking about starting to learn regex and this was really helpful as an introduction. Thank you.
Thank you Luke for this wonderful video.
7:32 listen carefully when he says "spool", he makes the perfect old Minecraft fall damage sound! Wizardry
What the frick guys
The Word of Luke have Power...
I was literally looking up several articles about regex today! This is perfect timing.
luke kept his knee in the face cam the entire time what a power move
Thank you for this Luke! This is a HUGE help for me 👍
Just in time, I needed this for writing my first script, thanks Luke!
Thank you ! It is a great lesson!!!
thank you for video, i recently have been trying out DWM with dwmblocks, and wanting to write scripts for the statusbar, this was very helpful. im finding shell scripting very interesting and fun. i sure want to learn more. hope there's a follow up advanced video.
Suggestion for the future video: full-text instant search in a local 2 TB archive of textbooks and articles (PDF + DJVU). Using regex, of course.
insane
Thanks Luke. Good video!
Thank you Luke!
@Luke Smith keep up the good work 👍
Is there a reason you were using double quotes instead of single quotes? I would be more inclined to use single quotes with grep and egrep to avoid accidental expansion problems. Generally I only use doubles when I know that I really want expansion. Here is an example to demonstrate the difference. Add two lines to your rt file: The first is the sentence "Navigate to your $HOME directory." Then add the absolute address of your home directory to the file. You could do this with the command "echo $HOME >> rt"
Now the results of grep "$HOME" rt and grep '$HOME' rt give very different results.
Also, I tend to use either egrep or grep -E instead of just plain grep. This is in part because I cut my teeth on Regular Expressions in Perl, and egrep is closer to the Perl that I learned first. It is also considerably more powerful.
Thank you very much for this. Please make advanced video on this too if you get time.
No bloat on these pups. I like that.
Thanks Luke, super helpful! Cheers
Thanks for sharing
this was a useful video. ty luke.
I have to admit the oomer convention made me laugh and was very meta
now that we've found all instances of a certain thing in a text file, can you make a video on deleting, moving, replacing, etc. -- putting to use the output we've gotten here?
thanks a lot. chad-like teaching content as usual
Regular expressions is one of those things I need once in a while, wind up spending a bunch of time creating something that looks like I hit my head on the keyboard, then forget how I did it months later.
It can be simple, but it can also start to get long and tiresome if the requirement is more complicated, like matching any valid IP address
"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"
Luke shows us magic.
Yo I have no idea what I’m watching but it sounds cool.
Very useful :) thank you!
Спасибо! Хорошо всё объяснил. Хорошее видео
I wa implementing search functionality on my web app and regexp is essential thanks for this crash course
5:04 yeah it's from the regural expressions of formal languages (more specifically Regural languages which are equivalent of L3 languages, those that can be generated form a right-linear grammar) where x* means {x}* so basically (x^0, x^1, x^2, ... ) x^0 is epsilon or lambda (also known as the empty word, a word of 0 letters) and x^1=x, x^2=xx, x^3=xxx and so on
5:28 {x}^+ is basically the same as {x}^* but without the epsilon/lamda, so it's (x^1, x^2, x^3...) As you can see this have something to do with Math more specifically with monoids. More infos here: www.ncbi.nlm.nih.gov/pmc/articles/PMC3367686/ en.wikipedia.org/wiki/Regular_grammar
Vim diesel
I think newbies out there would also like to know that
and \t also have special meanings: end of line and tab respectively. I'm not sure if that works with grep but it does work with other tools like python.
I use them all the time
Hey Luke, sorry if this is too personal, but I noticed that you've started to display a lisp. Did you recently get Invisalign braces (or has the social isolation lowered your pronunciation level)? I had braces put on in my mid 20s and they made me a little lispy too
Good video. Maybe make a part 2 where you go over grouping and such, as it is also quiet important :P
"[a-Z]" should work the same as "[A-Za-z]".
Although there might be a difference I'm unaware of.
It matches by the actual code of the characters, and there are characters between z and A, so it would match all the letters plus other stuff.
It will probably behave in an unexpected way, because the upper case letters come BEFORE the lower case letters in ASCII...
@@elandje I was going to answer this, but if you try grep with [a-Z] it works, but if you do it with [A-z] it says "invalid range end". What??
@@FyahBurn95 It is like I said, 'Z' comes before 'a' therefore it is invalid. [A-z] works, but includes a few non-letters, like [, |, ] and @. Search online for ASCII table.
@@elandje Have you tried it? [A-z] does not work with grep for me, but [a-Z] does. I know what the ASCII table is and the fact that A-Z comes before a-z, and also that UTF-8 is an extension of ASCII, which is what matters unless you actually work with ASCII files.
Luke Smith has saved Linux overnight, invented time travel, an alter ego and global warming.... and of course LARBS ;-)
instead of [0-9] for all digits you can use \d, similarly for any non-digits you can use \D
Love regex so much..
Me thinking yesterday: I should get around to learning regex for better grep/sed/awk etc
Then you put this up. Cheers uncle Luke
Luke, have you considered creating videos on cloud? You appear to be a systems thinker and I’d bet that you can teach cloud technologies pretty well.
Tnx for regx!
Could you make a video on git and show us how you use it on your daily basis?
Great video. Thanks for sharing these fundamentals of regular expressions. Do these basics work on Vim? Thanks again.
Good introduction with good examples! To throw something useful into it, don't you find weird you need to escape '+' but not '*'? And '*' is a (famous) shell expansion, but '+' is not! Also, if this was the case, it would be solved by the usage of double quotes you do, which avoids expanding anything but the dollar sign (for variables), or by switching to single quotes, which don't allow for any shell processing. The actual thing happening is that grep uses default regular expressions and the plus sign is from extended regular expressions, which grep understands but only if they are escaped. To use extended regexps without the need for escaping their metacharacters, try egrep or grep -E.
The "+" was not part of the original set of special characters; in fact, "a\+" (or "a+" with EREs) is just an alternative way to write "aa*".
Nice work. I like your videos on general command line tools. Can I suggest a presentation on sed or awk? I know these tools may require longer videos but I m sure you can manage it
wow that's a regex tutorial now.. can't wait more deeper examples
Will it help me find a soulmate?
yes
@@JurajOravecSGOrava y
y
y
y
y
y
y
y
y
y
y
...
Really helpful.
I was going to Email you to ask you to do this video, Thank you Luke very cool.
9:20 Finally, I can call someone every single oomer at once
I consider myself decent at bash but your videos always provide value, thanks boomer
How are you quickly saving the 'note' file? I know ZZ is similar to :wq, but what's the similar command to :w? Where would I find the docs to read about commands like ZZ?
If you decide to make a more advanced tutorial definitely include lookahead and lookbehind as I use them all the time.
luke how did your setup handle multiple languages for your linguist work?
i'm also interested in this, as i use arabic and spanish.
Regex is awesome.
doing gods work! any cool projects like the corona project you had before?
This will actually land me a better job.. wow.
to match any letter, lower case or upper, i think you can do [a-Z]. much better.
I found this really usefull
boomer, zoomer, doomer, coomer. I can' t keep up anymore, I must be gbetting old. Apparently I am a boomer now according to the zoomers even though I never was a boomer before.
boomer is a mindset as well as a generation of people that destroyed the USA
Luke before: Searches for Jesus
Luke now: "Crucifixion, well that's a nice thing as well"
What I miss in this video: What to *actually* use regex for.
Here's what I've used regex for:
Refactoring code - I had some functions in JS that I wanted to turn into lambda's.
Replacing HTML in several files - The files were partially identical and II wanted to replace something in the header elements.
Find non-ascii letters - I copied Haskell code from a book and it didn't compile, because the book used unicode quotes for the comments and GHC broke on those quotes.
Find the nth comma in a CSV file - I wanted to remove everything after the 3rd comma or something like that, because I didn't need that data - the file was too large to open in Excel.
At the end of the day regex is a tool to serve a purpose. Don't learn it for the heck of it. Learn it because you can use it to solve problems.
wouldn't cut be better for the CSV thing? good tip for finding non-ascii though, that will come in handy.
@@der0keks Huh, didn't know the cut command was a thing! I'm more of a Windows guy, so I'm woefully behind on my terminal knowledge.
Happy to hear someone has found something useful! The regex was [^\x00-\x7F] BTW, which is basically searching for any NOT ASCII char.
Will you be able to design a regular expression which matches exactly the words a^n.b^n (a random number of 'a' followed by the same number of 'b') ? ;-)
So the criteria is that it must match instances of ab where the number of a's and b's are equal? So ab or aabb would match but abb or aaaaabb would not match?
@@magetaaaaaa yes this is the criteria :-)
@@thefantasicm_2407 Hmmmm, I'm not a regex guru by any means but I feel like there would almost have to be something out of regex to do the analysis? I feel like trying to come up with a solution now.
@@magetaaaaaa This is a tricky question : the answer is that it is impossible to recognize this language with regular expressions, sorry :-). This is related to the Kleene theorem. Look for Pumping lemma (regular languages) to understand why.
@@thefantasicm_2407 Hmmmm, maybe something like this would do the trick with Python.
import re
file = open("textfile.txt", "r")
for line in file:
a = line.count('a')
b = line.count('b')
if re.match("^a+b+$", line):
if a == b:
print(line)
I've yet to see a more digestible explanation of regex. And I have seen many.
When is the jagex tutorial coming?
I clicked on this video because from the thumbnail I thought Luke was naked (his T-shirt colour).
Awesome. Next awk?
I never expected that GG Allin would lecture me about regex someday.
What is a file containing images of minors called?
Or you can use \w to match letter and \W to match non letter
And \d to match digit and \D to match non digit
CMIIW
hey luke How can I integrate uganda.txt to my neovim i reaaally like it
Ed is the standard text editor
You match with regex but how you can extract ONLY the matches? For example from a text file like your I eant to match and print out ONLY the emails , how can you do it?
We can use sed.
@@bahathir_ can you show me an example?
@@GooogleGoglee
Actually it is quite compicated. There are many patterns to be matched .. Try one of the solution in this URL.
Thank you.
www.unix.com/shell-programming-and-scripting/181361-sed-regex-extract-email-address.html
You can use -o option of grep
grep -o your-regex file-name
Congrats on reaching 90k subs Luke. Waiting for the 100k special video where you install FreeBSD on a Thinkpad X60.
Tip for anyone like me out there:
Watch all these videos at 0.75x.
😉
haha
You read my fucking mind
This actually is just in time for my sys admin class at uni over the summer, thanks luke!
Finally a regex to human translator
* means actually anything anytimes, right? I mean, you don't need to write .*
Also, you've said that * means any number of previous character, but, in fact, it doesn't have to be specifically the previous character, it can be any charactar.
Can someone tell if I'm right or wrong? Because I have learned that regex a couple of years ago and, since then, I use them the way I describe.
In the shell, and I'm sure in some other places, * does mean any character but I think the standard
Regex grep uses, the basic and the extended one as well as PCRE, treats it as _any number of_ so you
would have to say .* to match any character
When I use grep it only shows the line without highlighting the matching text. Am I missing something?
Yes, the --color flag
My regex: oh, this makes sense
Someone else regex: do i really need regex if i use 500 greps and a string concatenation?
15:30 TLDs actually got more complicated than that: en.wikipedia.org/wiki/Internationalized_country_code_top-level_domain
My life: This is a learning exercise. Who cares.
Luke could you make video about history of linux terminal editor ...