At 8min, when you pipe uniq, note that 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use 'sort -u' without 'uniq'. so it would be | sort | uniq OR | sort -u
This is a great tutorial about awk. I've learned quite a bit. The only thing that red flagged me was that NONE of those were square roots. They were squares.
I just came to say that I can see that you listened to the comments and now you are sharing your knowledge on simple but technical stuff that's really helpfull. I apreciate that and also can feel that effort
I tried it with sqrt(i) and that worked. I kind of wonder how much math you can do with awk... Yes, exp(var), log(var), cos(var), sin(var) - but not tan(var); var**x gave exponent, as did var^x. Lots of different functions...
I have always had trouble with awk because of all the syntax involved, this helped sooo much. When you pressed enter on the square root part then said that the output was correct, this is the task list I had to perform to get back on track: 1. Facepalm 2. Pull fire alarm 3. Crying session 4. Shower with clothes on 5. Call Mom 6. Eat some sweets 7. Finish the video
Good introduction to awk. Back in 2013 i had to parse huge log files that had no field separator, even worst each line could be completely different depending on certain conditions. Awk saved my life.
The problem with awk is that most people know it but all seem to think its whole purpose is just to extract columns. This Video gives many practical example that it's so much more.
I have been using linux for years living on a few commands. I had only a faint knowledge of the command/programs such as awk or sed. Mainly because I had no ideas of their capabilities and use cases. Now I can see how useful they can be. I usually recommend your channel to my PhD students and observed a real improvement in their computer skills. IMO your channel is vital for linux wannabe and "familiar but not expert" users. So thank for your work.
Thank you Derek I appreciate the time and effort you put into making these videos. You help us "middle users" take things up a notch. Middle as in we are no longer beginners but won't ever be sysadmins. We do however want more out of their Linux system.
This is great, this video arrived just in time since currently I am on problem automating a lot of stuff with bash scripting and using awk for text-processing. Thanks for the content!
Great video. An accessibility note: Your console’s active line is at the very bottom of the screen, which is where UA-cam on Smart TVs puts the subtitles, so the subs block the thing you’re typing. It would be better if your terminal was sized so that the bottom 10-20% of the screen isn’t used.
yeah, that was one example that did not resonate at all because wc will also provide totals when applied against multiple files - much less to type to just use wc
Good video. Some of these, I didn't knew. An error in the video: it wasn't the "square root", but just the "square", or "power of two". The square root of 4 is 2, the square root of 9 is 3, the square root of 25 is 5, and so on.
yes because obviously all these 40 year old Unix tools are buggy and broken after thousands upon thousands of users applying them daily over many decades
Well. People have already rewritten it in languages much less speedy than Rust - Go and even Common LISP. Knowing this, rewriting it in Rust is a much better idea !
A little bit of Awk can be useful. My standard example: I have a list of shares I track. The website I use lets me download data, but holds more data (open, close, max, min,volume ) than I want. Awk is perfect for filtering a single column (close values).
small suggestion, bring the bottom of your terminal window up a bit so the youtube progress bar doesn't cover up the command you're entering when the video is paused. Thank you for making this!
I know in most window managers you can change the size of a window from left or right but not on the top or bottom at least from what I have seen unless you are in floating mode than you can change it to any size. Most of the space around each window is configured hard coded in your window manager config. You could change it but than would have to change it back after a video.
Yup, it's a complete language, as it is taught in CS degree courses (when taught). However this guy, otherwise, is a far more interesting teacher. Thanks, DT! This app/language confusion here is, like with Python, there is the language, AND there is ERIC, a free IDE that comes with Python. ERIC executes Python, but it's NOT Python. The AWK IDE is more invisible, but there. You can write independently, then execute.
A couple months ago I wrote a little shell script to gather system information such as CPU model, memory, computer model, shell, WM (using lscpu, dmidecode, etc) and I had to learn a little bit of Awk, Sed, and Grep to clean it up and make it look pretty. This video would have came in handy back then, instead of spending 30 minutes reading StackExchange articles on them!
I think a very good example of how good & useful is awk is the example of a log like the apache access log (in the default format) and properly filter the requests that responded with a 400 or 503 response, given that you can find 400 and 503 in the request address, or in the size or the time it took to complete. Also to filter all requests that are longer than a second for example and many other things like that. It's basically Excel powers in CLI over column-structured data/files (which are common in commands outputs or log files)
Yup, awk is cool. Just as an aside that was squares, not square roots. For square roosts you would need something like (n^(1/2)). AWK does have ^ which is raise to the power function. Raising a number to a fractional power is taking the root.
Thanks for the share. I used use text editors like Textpad to do these things with regex, but this is so much better since not just find/replace, but perform operations on each result, and being able to easy have in a shell pipeline of commands. Always too lazy to learn it for some reason, but now I'm motivated.
Hey DT. Great review of the power of "awk". You cover all the basic functions for a new user. I use "awk" in a shell script to sort through my datalog's results that i need. Then dump it to a csv file.
Thanks for making this video! I've tried a couple of times to add AWK to my arsenal, but gave up halfway. This has inspired me to give it a more serious go. I'm generally pretty comfortable with grep and sed but looks like I can make this work for replacing those two for many use cases.
I really need to push myself to use awk more. A lot of this can be done by just combining grep, sed, and cut, but it’s kinda nice having it in one process.
Yea, you can actually use "awk" as a scripting language. Might have been useful in past decades. But now, I'm not sure why anyone would script with awk when things like bash and python exist.
To print the last words or specific words we can use the cut -d "." -f3 or -f3- -f3 prints only 3rd word -f3- prints rest of the strings including 3rd word
First: a good video, thanks! But it might need a "part ii", going deeper into what makes awk special among other languages. Awk is what at least I call a "pattern-action" language. So, the thing before the braces is a pattern, a kind of conditional structure telling for which input rows to execute that given block of code, and the block of code is then what to do when the pattern does match. Pattern here refers to a much wider set of conditions than just regular expressions; it can e.g. be a comparison against some variable you set yourself within the program, based on some previous data, and so on. So, if your statement within the braces is just "if (some condition based on the input row)", you can just lose the "if" and move the condition outside the braces, to have the same effect. You also briefly show that the pattern can specify a range of rows, by having two expressions to match the first and last rows of the data to process for that particular action. This could be for example "/start-marker/,/^$/" to process all sets of rows that begin with a row containing a "start-marker" and end with an empty row. Whatever comes after the empty row before the next "start-marker" is ignored (by this action, but you can have other actions in the script to do something else for those rows). The patterns for the beginning and end of the range do not need to have any relation to each other. These together make ti possible to very rapidly build "state machines" for processing text with some more complex but still regular structure (IBM JVM garbage collection logs being one example of such data format; I've done some awk pieces that pull out numeric data out of those and then output that data in a "only numbers" CSV form that can be pushed out to a spreadsheet for producing graphs out of the data). It is of course possible to do the same with any general-purpose language, such as Python or perl, but they don't lend themselves quite as well to building such processing state machines as awk does, it takes much more time to have the equivalent Python/perl code look anywhere as neat and readable.
@@swarooprajpurohit110 well, it very much depends (on what level of learning and relevance to you personally too). Example: I know awkward, sed, etc exist. And if required, I could write a complete program in PERL, (which eats awk for breakfast, because that is exactly WHY PERL was devised...) to solve a problem. In reality, I'm usually solving problems at a much simpler, "high" level. So, you can position yourself in the "space".... I cannot overstate the relevance of experience. Not necessarily in coding, but in EVERYTHING.
@@swarooprajpurohit110 just start by learning to completely understand the top display. top is like task manager. Then you could read the man pages for the main utilities. Learn a bit about how the cron demon works and what it is for. From there just install a few different distros and find your favourite.
I am actually going through the famous Awk, the programming language for the last two weeks. And, I spotted a least 1 verboseness. When you are trying to format the output of your shells, you don't need the regular expression to look for slashes. Not sure why you did that. It might also be helpful to people to know that you are turning NF(which counts and stores the number of fields) into a $1, $2, however many fields are in each line, that is what $NF becomes. If it is 4 fields, then $NF becomes $4 and so on.
the -i option is a huge plus for sed because can just do mass text search and replace operations on mass number of files all at one go, right from the command line
Dude, plz do Udemy or any full courses, I came here from some lame udemy course, because there was no deep dive on awk command like here love Ur content, awesome !
AWK is not just a text processing utility, awk is a complete programming language!!
yes, remove Perl, Python, and Node JS (notice no mention of Ruby - ah how far they do fall) - awk and bash is all you need
@@TheSulross you need Perl to keep your sanity, you can remove the rest
Calm down. It's not that exciting.
@@gordongoodman8342 it is, a lot!
yup! I was going to comment the same thing. you could even use it as your shell.
At 8min, when you pipe uniq, note that 'uniq' does not detect repeated lines unless they are adjacent.
You may want to sort the input first, or use 'sort -u' without 'uniq'.
so it would be | sort | uniq OR | sort -u
indeed
quite
rather
indubitably
moist
This is a great tutorial about awk. I've learned quite a bit. The only thing that red flagged me was that NONE of those were square roots. They were squares.
I just posted this same thing, before seeing you posted it.
He just got the order backwards. "The square root of i*i is i". Anyone could have made that mistake.
I just came to say that I can see that you listened to the comments and now you are sharing your knowledge on simple but technical stuff that's really helpfull. I apreciate that and also can feel that effort
Had to process a 120k line csv at work the other day. Awk was a god send
DT: AWK is important for linux users
Luke Smith 1 hour later: Why AWK is useless for linux users
_Awk is the predictable result of misguided Enlightenment philosophies._ 🤭
Brodie 2 hours later: Boomers fighting over usefulness of linux utilities
_Why learning awk when you can learn grep_
Luke Smith 1 hour later: "You can replace Awk with Emacs"
10hs later: "You can replace your wife with Emacs"
@@MicroClases_Ciencia 10.5 hours later...I did. 😂
That is the most clear demonstration of AWK I have ever seen. Amazing.
Learned more about awk in 20 minutes than in the previous 40 years
awk is a must. So powerful.
Btw. You actually did just square. Not square root :)
Could have done the square root of i*i is i
I tried it with sqrt(i) and that worked. I kind of wonder how much math you can do with awk... Yes, exp(var), log(var), cos(var), sin(var) - but not tan(var); var**x gave exponent, as did var^x. Lots of different functions...
I have poorest knowledge of maths but I paused and rushed to comment section to check I am not wrong. But tutorial is very good.
I have always had trouble with awk because of all the syntax involved, this helped sooo much.
When you pressed enter on the square root part then said that the output was correct, this is the task list I had to perform to get back on track:
1. Facepalm
2. Pull fire alarm
3. Crying session
4. Shower with clothes on
5. Call Mom
6. Eat some sweets
7. Finish the video
Good introduction to awk. Back in 2013 i had to parse huge log files that had no field separator, even worst each line could be completely different depending on certain conditions. Awk saved my life.
In the square root example, very time DT said square root, he means square!
Great video, DT!
Just swap the 2 fields and pretend you didn't see that:
"10 is the square root of 100" is a completely valid statement 😃
@@antoninperonnet6138
I could do that! 😉
It's AWK math
@@KeithPeters 🤣🤣🤣🤣
The problem with awk is that most people know it but all seem to think its whole purpose is just to extract columns. This Video gives many practical example that it's so much more.
14:00 - /^[b,c]/ matches anything that begins with a "b" or a "c" *or a comma*.
neat
So true.
Regex yeet
Glad you made this video. I was stumbling upon awk in everywhere and wanted to learn it but didn't have time to do so. Thanks
This is one of the best explanations of awk I've seen! I'll save this one for some people I know that are still just learning Linux.
I have been using linux for years living on a few commands. I had only a faint knowledge of the command/programs such as awk or sed. Mainly because I had no ideas of their capabilities and use cases. Now I can see how useful they can be. I usually recommend your channel to my PhD students and observed a real improvement in their computer skills. IMO your channel is vital for linux wannabe and "familiar but not expert" users. So thank for your work.
He also inspires me to learn about this powerful utilities.
Nice, thank you. I am beginning my journey on bash and this is awesome.
Great tutorial. I learned more in 20 mins w/ awk then in 7 years in IT.
What did you do then in 7 years of IT? :))
Thank you Derek I appreciate the time and effort you put into making these videos. You help us "middle users" take things up a notch. Middle as in we are no longer beginners but won't ever be sysadmins. We do however want more out of their Linux system.
This is great, this video arrived just in time since currently I am on problem automating a lot of stuff with bash scripting and using awk for text-processing. Thanks for the content!
Great tutorial DT! AWK is very powerful!
Thanks, Josh!
@Earthling-Z3R0 Thanks man. You're right.
Great TUT. I ran into an issue because I don't use uniq and sort often enough; uniq doesn't work if dupes are not adjacent so you need to sort first.
I love you man, you're the only one who was able to explain awk in a simple yet effective way. I finally understand how awk works!
Great video. An accessibility note: Your console’s active line is at the very bottom of the screen, which is where UA-cam on Smart TVs puts the subtitles, so the subs block the thing you’re typing. It would be better if your terminal was sized so that the bottom 10-20% of the screen isn’t used.
The simple fact that fish is your default shell deserves a like and a subscription.
Sir, your a wizard with awk, i only knew half of this 😂
awk is awesome, but if you just want the number of lines, using wc is less characters to type :)
yeah, that was one example that did not resonate at all because wc will also provide totals when applied against multiple files - much less to type to just use wc
The point is to show what you can do with awk =)
Good video. Some of these, I didn't knew.
An error in the video: it wasn't the "square root", but just the "square", or "power of two". The square root of 4 is 2, the square root of 9 is 3, the square root of 25 is 5, and so on.
i came to the comments expecting every comment to be this. I am highly disappointed in the YT comments shit posting today.
@@christopherdavies7403 And this guy says 'I didn't knew', grammar mistake.
@@gordonbai5320 ok boomer
@@charlieoocharlie9466
FLAME WAR!
And you live in your mother's basement.
I learned so much more here in 20 mins then I did in my last two classes about this
Needed a quick refresher on awk as I had no used it in years. This was a great video, covering the essentials. Cheers.
I bet someone is already rewriting awk in rust for the memes.
you mean for speed and safety? /s
@@alessandroferrari4699 people use rust because it's fast and easier to solve errors and even rivals c in speed while being easier.
yes because obviously all these 40 year old Unix tools are buggy and broken after thousands upon thousands of users applying them daily over many decades
@@TheSulross exactly 😂😂
Well. People have already rewritten it in languages much less speedy than Rust - Go and even Common LISP. Knowing this, rewriting it in Rust is a much better idea !
A little bit of Awk can be useful.
My standard example: I have a list of shares I track. The website I use lets me download data, but holds more data (open, close, max, min,volume ) than I want.
Awk is perfect for filtering a single column (close values).
small suggestion, bring the bottom of your terminal window up a bit so the youtube progress bar doesn't cover up the command you're entering when the video is paused. Thank you for making this!
I know in most window managers you can change the size of a window from left or right but not on the top or bottom at least from what I have seen unless you are in floating mode than you can change it to any size. Most of the space around each window is configured hard coded in your window manager config. You could change it but than would have to change it back after a video.
@@warhawk_yt clearly he just needs to increase the edge padding to something crazy like 48-64 pixels for the [ M A X I M U M R I C E ]
This is truly amazing, hope to have another new Awk video in the furure!
DT, you are amazing man. Yours are the only videos I have been watching on a daily basis. Thanks for sharing all the knowledge!
Hell yeah!!! AWK arrived at the right time!! Thank you Derek!
Heaviest 20 minutes in my life. Worth it
Thanks buddy for such clear explanations. Now I'm comfortable to start write some crazy scripts.
I have been putting this off for a long time thanks for encouraging me to finally learn it.
Great video DT! Thanks for the awesome awk tutorial
Except:
s/square\ root/square 😀
My payment will be available in 2 weeks, then I'll be a patreon. Your content is amazing.
Yup, it's a complete language, as it is taught in CS degree courses (when taught). However this guy, otherwise, is a far more interesting teacher. Thanks, DT! This app/language confusion here is, like with Python, there is the language, AND there is ERIC, a free IDE that comes with Python. ERIC executes Python, but it's NOT Python. The AWK IDE is more invisible, but there. You can write independently, then execute.
A couple months ago I wrote a little shell script to
gather system information such as
CPU model, memory, computer model, shell, WM
(using lscpu, dmidecode, etc)
and I had to learn a little bit of Awk, Sed, and Grep
to clean it up and make it look pretty.
This video would have came
in handy back then, instead of spending 30 minutes
reading StackExchange articles on them!
Could you share your script? I would like to use it!
Your audio sound 1000x better then before
Thanks for this video. It helps me understand how to use awk better.
I think a very good example of how good & useful is awk is the example of a log like the apache access log (in the default format) and properly filter the requests that responded with a 400 or 503 response, given that you can find 400 and 503 in the request address, or in the size or the time it took to complete. Also to filter all requests that are longer than a second for example and many other things like that. It's basically Excel powers in CLI over column-structured data/files (which are common in commands outputs or log files)
Thank you professor DT for the lecture .
Straight to the point & excellent as always. Keep up the good work DT ! 👏
This is really helpful and practical, thank you DT!
That was some awesome awk-fu bro, thank you!
love both Sed and Awk
Never needed awk in last 6-8 years or so... but thanks!
This is the kind of stuff I love to watch
This couldn't have come at a better time! I was seeing this command everyone in people's dwm configs and had no clue what it is.
Awk ... very meaningful name !!!
Looks like an assembly instruction.
Yup, awk is cool. Just as an aside that was squares, not square roots. For square roosts you would need something like (n^(1/2)). AWK does have ^ which is raise to the power function. Raising a number to a fractional power is taking the root.
3:00 `cat /etc/passwd | cut -d':' -f1` also works
DT, this is an excellent video! Thanks!
Thanks for the share. I used use text editors like Textpad to do these things with regex, but this is so much better since not just find/replace, but perform operations on each result, and being able to easy have in a shell pipeline of commands. Always too lazy to learn it for some reason, but now I'm motivated.
Hey DT. Great review of the power of "awk". You cover all the basic functions for a new user. I use "awk" in a shell script to sort through my datalog's results that i need. Then dump it to a csv file.
Excellent presentation!! to the point and straight forward! great job!
Very useful info DT! I only knew how to print columns in awk.
Very nice introduction to awk features. Thanks for sharing this useful knowledge!
Great introduction of `awk`
3:30 can be done with 'getent passwd' :-)
13:45 you print the square of, not square-root of.
Thanks for making this video! I've tried a couple of times to add AWK to my arsenal, but gave up halfway. This has inspired me to give it a more serious go. I'm generally pretty comfortable with grep and sed but looks like I can make this work for replacing those two for many use cases.
This is a great video on AWk..thanks
Great video, helpful comments and even better resourceful and corrective people in the comments
i always found it easy to use because i'm AWKward.
Nice Luther insult at 8:20! I for real laughed out loud on that one. I'm surprised no one called that out in another comment...😁
Great tutorial! I do prefer the WTF command. It covers all my confusion when trying to navigate the command line. 😜
This is a great review of some practical usages.
I really need to push myself to use awk more. A lot of this can be done by just combining grep, sed, and cut, but it’s kinda nice having it in one process.
Amazing! Basically I can programming using awk. Thanks
Yea, you can actually use "awk" as a scripting language. Might have been useful in past decades. But now, I'm not sure why anyone would script with awk when things like bash and python exist.
@@DistroTube maybe awk faster than python and more specific programming language bash or posix shell
To print the last words or specific words we can use the cut -d "." -f3 or -f3-
-f3 prints only 3rd word
-f3- prints rest of the strings including 3rd word
Cut(1) would be my go-to tool over awk if only they'd do their delimiter handling properly. Until then, it is rarely touched. So sad.
Awk is awesome, great vid I learned a lot
First: a good video, thanks!
But it might need a "part ii", going deeper into what makes awk special among other languages.
Awk is what at least I call a "pattern-action" language. So, the thing before the braces is a pattern, a kind of conditional structure telling for which input rows to execute that given block of code, and the block of code is then what to do when the pattern does match. Pattern here refers to a much wider set of conditions than just regular expressions; it can e.g. be a comparison against some variable you set yourself within the program, based on some previous data, and so on. So, if your statement within the braces is just "if (some condition based on the input row)", you can just lose the "if" and move the condition outside the braces, to have the same effect.
You also briefly show that the pattern can specify a range of rows, by having two expressions to match the first and last rows of the data to process for that particular action. This could be for example "/start-marker/,/^$/" to process all sets of rows that begin with a row containing a "start-marker" and end with an empty row. Whatever comes after the empty row before the next "start-marker" is ignored (by this action, but you can have other actions in the script to do something else for those rows). The patterns for the beginning and end of the range do not need to have any relation to each other.
These together make ti possible to very rapidly build "state machines" for processing text with some more complex but still regular structure (IBM JVM garbage collection logs being one example of such data format; I've done some awk pieces that pull out numeric data out of those and then output that data in a "only numbers" CSV form that can be pushed out to a spreadsheet for producing graphs out of the data).
It is of course possible to do the same with any general-purpose language, such as Python or perl, but they don't lend themselves quite as well to building such processing state machines as awk does, it takes much more time to have the equivalent Python/perl code look anywhere as neat and readable.
For pattern matching, try SNOBAL 😉
Hey, I'm a beginner. Can you tell me if I've to learn commands like awk, sed, etc or should I come back when I start bash scripting?
@@swarooprajpurohit110 well, it very much depends (on what level of learning and relevance to you personally too).
Example: I know awkward, sed, etc exist. And if required, I could write a complete program in PERL, (which eats awk for breakfast, because that is exactly WHY PERL was devised...) to solve a problem.
In reality, I'm usually solving problems at a much simpler, "high" level.
So, you can position yourself in the "space"....
I cannot overstate the relevance of experience. Not necessarily in coding, but in EVERYTHING.
@@Theineluctable_SOME_CANT Hey, I didn't get you. I meant what would your recommendation be for a beginner?
@@swarooprajpurohit110 just start by learning to completely understand the top display.
top is like task manager.
Then you could read the man pages for the main utilities. Learn a bit about how the cron demon works and what it is for.
From there just install a few different distros and find your favourite.
As I found out, you have to sort _before_ running the _uniq_ , otherwise it wouldn't work.
thanks awk command that help me a lot in my project after watching this video ^_^
What the heck?
That awk thing is amazing!!!!!!!!!!
Woa my life just changed
Commandline manipulation is such a lost art but more powerful than the wizards in LOTR.
04:40 min. Pipe the awk to column command. Like: | column -t
i really appreciate your effort , thank you for the great lesson MAESTRO .
Great vid DT!
you dont need to put the double quotes. you can also do
awk -F: '{print $1}'/ /etc/password
Please do jq tutorial next: it's so awesome to manipulate json data, combined with curl it's very powerful for working with REST apis
Is jq a GNU utility?
@@ghodawalamoamanzahirabbas8996 it isn't (according to short duckduckgo search :) ) - but it's amazing nonetheless!
Your example for the line count can be reproduced with the command "wc -l /foo/bar/file_a /foo/bar/file_b"
I am actually going through the famous Awk, the programming language for the last two weeks.
And, I spotted a least 1 verboseness. When you are trying to format the output of your shells, you don't need the regular expression to look for slashes. Not sure why you did that.
It might also be helpful to people to know that you are turning NF(which counts and stores the number of fields) into a $1, $2, however many fields are in each line, that is what $NF becomes. If it is 4 fields, then $NF becomes $4 and so on.
My favorite tutor
Thank you for doing this video. Very helpful.
David Draiman loves this program.
Awesome... good to know how versitile awk is.
that is a very cool podcast, very useful. thank you
Quality content 💯
Also, if you wanted to also do ‘sed’ I’m all for it.
the -i option is a huge plus for sed because can just do mass text search and replace operations on mass number of files all at one go, right from the command line
@@TheSulross It's sick! I love these tools. It's amazing that we get to play and use them for work and not have to pay a dime.
This was helpful…. Thanks to u and ur team
awk sed cut are very good to use
Dude, plz do Udemy or any full courses, I came here from some lame udemy course, because there was no deep dive on awk command like here
love Ur content, awesome !
you are Linux Moster ;-)
Thanks from Kurdistan