How The 'awk' Command Made Me A 10x Engineer
Вставка
- Опубліковано 10 гру 2020
- Learning to use the 'awk' command increased my productivity significantly. awk allows me to quickly manipulate and inspect data in text files. This is especially true for column-based data, such as .csv files. Since awk is a fully functional programming language, I can quickly use awk to solve problems that are too complicated for programs like grep or sed.
A text-based version of this video with all commands can be found here:
blog.robertelder.org/intro-to...
SOCIALS
----------------
Patreon: www.patreon.com/RobertElderSo...
Tiktok: / roberteldersoftware
Linkedin: / robert-elder-software
Blog: blog.robertelder.org/?...
Twitter: / roberteldersoft
Twitch: / roberteldersoftware
Github: github.com/RobertElderSoftware
Facebook: / roberteldersoftware
Instagram: / roberteldersoftware
Merch: store.robertelder.org/?...
the narrative that lead to the tutorial is so cute as it never happened.
I sure hope so, or someone needs to learn about spreadsheets.
it never happened indeed and a maximum of 4 lines of python can do this, that's why I avoid learning these tools
spreadsheet? is that like lasagna or vegemite or nutella,@@Bjawu ?
@@jazzochannel There *are* some similarities with vegemite, now that I thing about it. You don't want to use too much.
I feel like people just don’t get this guy’s sense of humor
Many moons ago I helped maintain what we thought might be the largest awk script ever created - it converted between the proprietary FORTRAN code for compilers by our competitors into our own proprietary FORTRAN code. Of course, monthly, if not weekly, we'd find something it did not convert properly, and it would have to be changed. Not only was it possibly the largest awk script, but it was the most modified awk script...
Fascinating! Do you remember how many lines (roughly) it was?
@@artemiasalina1860 It was well over a thousand lines.
Sounds awk ward to work with…. I’m sorry
@@bide7603It's not funny, it's a way of life...
I once took a little boy under my wing who thought that text manipulation equated to social skills. He was a little awk ward
The awk command is now my favorite Linux command
I avoided Awk for 15 years, but then I saw the light.
@@catsupchutney I have a year left of avoiding it like the plague, maybe. Then I'll give it a shot.
Utilities like awk are one of the reasons I love Unix. Everything is in a text file and the OS gives you the tools to parse and manipulate the contents.
I used to use awk every day about 30 years ago. A nice walk down the memory lane with this video.
memory lane? why did you stop?
@@jazzochannelmy work now doesn’t involve coding and data manipulation any more.
Thanks for sharing.
Good luck with your 1% raise!
Thank you!
@@RobertElderSoftware so... did you get the raise?
The solution to the original problem was a bit of overkill, but this was a good example for some teaching of base awk functionality. This is a more direct and shorter solution, only changing the F lines and printing all lines:
$ awk '$2=="F" {$1=sprintf("%.1f",($1-32)/1.8); $2="C";} 1' temps.csv
(the single "1" at the end is the condition, which evaluates as "true", evoking the default action: print -- nice shortcut if you mean only to change some lines but print all)
I like how there was a narrative style to the video. That's really engaging. You've earned a sub!
Great pace too.
I get that awk is a useful tool overall, but with your given example, why exactly would writing a program be time consuming?
i think it was just a poor attempt at a motivating example and a joke at the same time
the whole premise was quite funny
It would take as long to write in python as it would take learning awk. At least pre AI, now AI could generate the command in moments.
@@jeanhaley3051 i just tried writing that python script and measured how much time it took me (without trying to rush it), it took 1 minute. it might be faster with awk in the long run? but 1 minute is good enough for me
Even less code with Perl. And this example is not recommended at all. What happens when the data is changed. It all fails.
'awk', 'sed', and 'Perl'. All I ever need for scraping from log-files!
head and tail is also useful for a redneck like me :D
If you know Perl, do you still have a use for Awk? (I'm wondering if I should learn it, I already know Perl. From the looks of this video, I can do the same with Perl in maybe 20-30% more characters - not enough more to trigger learning it)
@@obiwankenobe3962 Sure. Back in the old Unix days you needed sed and awk to to text munging. Now I just use Perl for all of it. Perhaps more characters to type in, but much simpler syntax and easier to learn, IMO.
@@BytebroUK thx for confirming my suspicion
I enjoy using GREP and SED and I've been really wanting to learn how to use AWK. This video motivated me to do so, I'm gonna!! Also LOL, the end of the video made me laugh a lot. I love this!
Wow.. I know awk well, but what a presentation. This is an underrated video. Keep it up dude.
Thanks, will do!
Bro went from a 0x developer to a 0.5x dev
10x0 is still 0
what a hater, the video was great
It is but the title is wayy over the top
Thank you for describing the pattern-matching and default assumptions so thoroughly and illustrating each point with example code. Until today have never seen any awk tutorial that could get me up to speed so quickly as your video. Kudos!
Showing how awk can replace both the grep and sed commands was icing on your cake.
And your last sentence was a big flaming torch sitting atop the cake!
Get real- once you learn regex, and in my own world, specifically Perl regex, that stuff becomes fairly trivial. "Analysing log files from umpty-hundred jobs" used to be a chore but these days I can script a working solution in an hour or so. In Perl. Because that's the tool I'm familiar with.
This is an amazingly instructive video. Thank you.
Dude........ Your content is brilliant. You must become more known! We desperately need more Unix content on youtube.
@4:10 "As you see, awk is not quite this smart. And here you can see how awk isn't quite as good as a CSV parser. "
In GNU awk set the built-in FPAT variable. In your example: FPAT = "([^,]+)|(\"[^\"]+\")"
This is video is great and demystified awk in a very easy to understand way
Amazing presentation! I've been avoiding Awk for close to a decade, so much so I default to Sed for this kind of tasks! After watching your video, I feel more confident to try Awk next time :) Thank you!
so much content in just 10mins. amazing!
Awesome explanation. Thank you
I always thought awk was super hard to learn. 😭 Thanks for explaining it so well! 🙏
This was an exceptionally good tutorial. Thank you.
Finally and after so many years, I understand how 'awk' works. Thank you, Legend
This is such a great explainer of a tool i rarely used, but saw the potential, although i never dove in deep enough to learn it. Thank you!
At my past job where we used Linux exclusively, I used awk extensively along with bash scripts, sed, tr, od, etc. There are so many handy Linux commands.
"I was so unproductive!" → Proceeds to open the Gnome terminal
Alacritty FTW 🙂
This is why people hate the Linux community.
using a better terminal does t make you more productive, just pipe the massive prints so you dont have to worry about print speed on compilation or something. (i say this as someone who uses a fast terminal (foot))
Nice Video, this is a great explanation of awk, thank you for sharing.
Awk is a very powerful program indeed. The only thing that keeps it from completely replacing grep for me is the lack of a Perl mode for regex.
Before awk I was living on the street and had no significant other.
With awk I'm now living on a mansion and have two wives 🎉
Excellent explanation.
Nice tips and examples, man.
Greetings from Brazil.
Great video! So, it's 3 years later, I hope you got your juicy 1% raise!
I've never done more with awk than printing columns from a text file, like the 'cut' command. And for other things, use a perl one-liner. Loved the average calculation example in awk. Good to have a new tool in the belt!
I used awk and sed to convert entire flight manuals into our dataformat that won us the deal with pilatus as we could convert it to our software suit.
Excellent presentation of how AWK works and how it could benefit you.
I haven't used it personally (I would use Python: slightly more complex to use, but MUCH more powerful), but I know people who use it often, e.g. to scan log files for certain situations. It is great for iteratively improving the filters, browsing through the data.
The awk command is my favorite linux command. 😂😂😂 Great content! Keep going!
At a time when Excel had only 65000 lines, I used awk to condense big text data files to a manageable size used the associative array feature of awk.
If you’re working with same files over again, I suggest to include one sample line for each pattern because you may not recognise the line from the pattern after 2 months.
The Fahrenheit example I’d done in libreOffice. Import and so on.
There was a time where you had difficulties getting an awk that works on windows.
agreed - awk is awesome and i use it all the time - the most extreme example was a ~400 line script to analyze network files extracted from big integrated circuits but it’s great (and fast) at processing big text files - and it’s highly portable too since it’s installed on pretty much every unix/linux system dating back to the 1960s
Real world examples IME look something like this:
Some poorly documented program just spat out a noisy status message and produced a bunch of poorly named files. You want to parse the message to associate metadata with the output files, then rename the files with metadata as part of the name.
First pipe message into awk to separate into fields, then do simple transform in awk or pipe into sed or grep, then sort file names for association with the metadata output. Then pipe both into awk again to generate a shell script to rename the files.
I love how simple programs developed 50 years ago are still extremely useful.
That is because the core of Computing hasn't changed much at all over that time, and UNIX did an excellent job of capturing that core.
I pity the fools who know nothing outside of a GUI. Because with a GUI, you get a single tool that can do only what its designers allow you to do. But the UNIX concept is to have a load of little tools that one can combine to do exactly that what you want it to do.
Perhaps I should write a GUI overlay for the shell, allowing the old UNIX tools to be configured and linked together graphically. That would be fun, and might help people discover the tools that are available at the CLI.
I like your shorts "this is my favorite command", but the longer are more explanatory and better pauseable aswell. Thanx
Hello, thanks for sharing this knowledge. Interesting way of presentation and very concise and informative.
The temperature data format is a good one for exercising awk. This is a data driven world.
Great vid thanks! @10:20 the "C" doesn't appear in the output. Should it be inside the double quotes? ie ` C" = "` be `" C = "`?
Good catch, you're exactly right!
My man ! Thank you
Super Useful, and entertaining
super helpful
I really think we need better parsing libraries and at some point the most effective way to dev will be an editor that is easy to add features to which essentially will be coding from the command line. Even between awk and sed there are editing cases where I cannot cleanly implement my operation/macro.
Neovim?
very nice video .thanks!
awk is amazing , I love it !
Humorous but informative, I'm no longer afraid of awk, nice video
Don't even ask how many python scripts i wrote to do this kind of stuff, grappling with pandas csv reader and such
BEST INTRO TO AWK. PERIOD!
I remember when I used to torture myself with arcane bash commands and construct a towering awk command for complex file parsing. The cryptic awk errors were chef's kiss.
Honestly I just prefer any other language at that point.
I was hoping there was a "but" to the "doesn't parse quotes" problem. it feels like I would always be running into the issue that I couldn't quite parse a file with 100% accuracy. (some line or other would be inconsistent somewhere.)
I challenged a guy who believes in awk, I let him define the problem. He could use awk, sed, … and I would use cpp, yacc, and lex. The challenge was on two fronts, dev time - first to start, and run time. He insisted that he may run slower than me, but he would clearly win dev time.
Well, I ended up not using yacc, just cpp and lex. Beat him in dev time by nearly an hour. Mine ran in less than a second and his took a few minutes.
To me awk is awkful with a silent K. It is a very useful tool, but it is just so easy to create your own specific, and much better tool quicker than using awkful.
You are the great teacher
Great channel
The awk command is my favorite command.
Awk is great, but not the only hammer for every nail. Sed, cut, bc, (node maybe), vim macro, bash, lots of ways to solve this with one line or so
I love this video. Awk is one of the basic UNIX tools I always skipped over.
Bad call, lol.
Question. In your initial use case, why is awk preferred over sed?
'sed' is generally preferable in situations where the text changes you want to make can be described by a regular expression. 'awk' is preferable in situations where the changes you want to make require more arbitrary computation that can't be described by a regular expression (such as a mathematical calculation). If you want to try and precisely describe the class of problems that each tool solves, you could probably draw a relationship to the Chomsky hierarchy:
en.wikipedia.org/wiki/Chomsky_hierarchy
However, describing the difference this way is a bit of an oversimplification that would be a good of starting arguments among computer scientists.
Ah the classic 10x engineer,
Making 10x the work for everyone else. I too have earned this little.
I enjoyed your presentation on awk and I subscribed to this channel. I have a question: Can awk to the following job -- send the first line of a file1, delimited by default space, to another file2, file2 that would take its name from the first few fields of the line 1 just transferred from the file1? Thank you.
Given a file 'a.csv' with contents:
field1 field2 field3
a b c
d e f
g h i
You can do
awk -F' ' 'NR==1{print $0 > $1"_"$2".csv";}' a.csv
which produces this file:
field1_field2.csv
with contents:
field1 field2 field3
OR what you maybe also wanted to do:
is this:
awk -F' ' 'NR==1{fname=$1"_"$2".csv";}NR>1{print $0 > fname;}' a.csv
which produces a file named:
field1_field2.csv
with these contents:
a b c
d e f
g h i
I just did this on my machine using GNU awk. Not sure if that'll work the same in all POSIX awk implementations though.
@@RobertElderSoftware Super! Thank you. I used based on your awk: awk -F' ' '{fname=$1" "$2;}NR>0{print $0 > fname;}' file
Writing a dedicated program would have taken about 7 lines of code and 5 min, not forever. But, I digress. Us Nix folks live by grep, sed, awk, cut, xargs, etc.
Thank you for saving my life!
;)
I use awk over grep daily because of it's usefulness in creating conversions or finding ways of pattern matching smartly. It's grep and cut in one. The only time i use cut is for specific columns due field separators being a pain sometimes.
Dude, this video was sick! Looovely stuff. You're awesome: 👑
yep, awk changed my habits either, thanks for sharing!
It's been 3 years since you posted the video. I hope you got that raise. Gotta offset inflation somehow.
Hmm, first time I watched this video, sound was behind the image, so you looked like Bruce Lee speaking in movies 😁 I thought it was intentional at first, then realized it was a problem in my browser. AWKward.
But, how would you use awk to handle csvs?
writing a dedicated program takes forever? I agree awk does faster, but writing a python script will take at most half a minute...
I'll never use IDEs and Vscode anymore to process texts!
Thanks ❤
For reference, here is a PowerShell solution for that problem:
Import-Csv .\temps.csv -Delimiter "`t" | Select-Object @{Name="temp";Expression={if($_.unit -eq "F") {"$([math]::Round(($_.temp-32)/1.8,1))"} else {$_.temp}}}, @{Name="unit";Expression={"C"}} | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")}
However normally you wouldn't do that as one line in PowerShell. You can just save the whole table as a variable and work on that instead:
$temps = Import-Csv .\temps.csv -Delimiter "`t"
foreach($entry in $temps) {if($entry.unit -eq "F") { $entry.unit="C"; $entry.temp="$([math]::Round(($entry.temp-32)/1.8,1))" }}
$temps | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | %{$_.toString().Replace("`"", "")}
Assuming you need it in exactly the same format again. I'd argue that using commas in a csv file and quoting the values would be better though.
A question.
If awk makes me 10x engineer and let's say grep makes me 3x engineer - does it mean output from this pipe:
$ awk {print $1} somefile.txt | grep somepattern
makes me 30x engineer? :P
I think you're really onto something here.
Unbearable.
Don't waste available tools: cat into awk!
@@andreipopescu7636 That's cat abuse. As a long-time guardian of cats, that bothers me.
Great presentation style. I just subscribed, hoping to hear if you got that 1% raise.
I didn't get any raise this year, but I've got a good feeling that if I work overtime for just a couple more years, someone will finally notice my hard work :)
Quick question:
@ 2:03 You added " ? " and " : " inside the condition parentheses.
Can you please explain what the question mark " ? " and the colon " : " are used for?
The '?' and ':' symbols are part of a feature that is available in most programming languages for doing in-line conditional assignments. It's usually called a 'ternary operator'. Here is some awk-specific documentation on this feature: www.gnu.org/software/gawk/manual/html_node/Conditional-Exp.html
@@RobertElderSoftware Oh.. now I understand. It is just like this... ( ? : )
@Nizar, yes, you are right. However, most programming language do not support the "if-condition-is-true" as an equation. For example, maybe it is illegal to do "isLoli? legal = false : legal = true".
@@NizarAjeebi Yep. From 'man gawk': "?: The C conditional expression. This has the form expr1 ? expr2 : expr3. If expr1 is true, the value of the expression is expr2, otherwise it is expr3. Only one of expr2 and expr3 is evaluated"
Its now 3 years later, he better have gotten that 1% raise
thank u
Thanks -- You have a new Sub!! Take care
Today AWK isn't only a text-processing utility, it is a complete general purpose programming language! You can use loops, variables, regex, conditionals, functions, array and string manipulation, math lib, include files...
Writing a script in C#, F# or Python to do this takes 1 minute and the code will be more understandable and can be copied into your project. Awk is Awesome in niche situations, but I'd only use it when other languages are not available.
yeah given this is just a backstory case for a awk lesson.
It still really depends the tech the dev is more used to. If the peson uses shell commands a lot everyday, it's easy to just awk, great to use what it offers. Otherwise is just no-brainer to write a small standalone app/script in your fav language (or any tech), hell it's most of the times just easier to google spreadseet or prompt into ai assistant (that appeared after this video being posted)
How about using Excel or Libre Office instead to open the csv file?
For just viewing a more complex csv, sure, but the transformations (especially more complex ones) are better scripted.
Libreoffice beats Excel for me with imports as it defaults to UTF.
nice video haha. hope you make more
The Awk command is my favorite Linux command
This is my favorite comment.
It's 3 years later, did you get that raise bro?
0:30 dedicated Python code for that task is 6 lines.
``python
with open('temps.txt', 'r') as input_file:
with open('temps_all_c.txt', 'w') as output_file:
temp, scale = line.strip().split()
if scale == 'F':
temp = (float(temp) - 32) * 5/9
out_file.write(f'{temp} C
')
```
There is also jq for json manipulation, but I've never used it in practice.
If you're working in a Windows environment, you can use PowerShell in much the same way for data processing tasks like this.
That said, I really need to start using awk in Linux instead of just contorting sed into doing things it wasn't made to do.
Perl was written because Larry Wall was frustrated with the Awk+Sed combo. Awk+Sed didn't do what he wanted it to do. Perl is still much better then the Awk+Sed combo, however very few people learn Perl but have reverted to learning Awk+Sed instead for text processing. Its a real shame because Perl is still hands down the best for text processing, whether it be on the command line or processing text files.
in 2024 it's counter-productive to rely on awk as it's missing libraries to work with common formats: multiline rows, binary (like parquet), csv with quotes, headers, footers, excel
i.e. pandas.read_csv in jupyter/ipython would do the same (could have a one-liner if you wish) but it's compatible with binary formats and anything you could possibly think of.
another option is excel if there is no need to repeat the transformation.
I tried awk as well but only managed to be a 3x developer, how do I get the extra 7x?
Its multiplicative I guess. So the question would be where he got the extra 3.33x
Hi, thanks for this useful video. Can you kindly explain what is the "%c
" modifier doing on the printf command ? thanks!
'%c' is a pretty common format specifier for 'printf' functions, although after looking into this specifically for awk, I don't think '%c' is officially supported by awk. Therefore, this is probably a non-standard feature and should therefore be avoided. Here is a general wikipedia article on printf format specifiers for C programming:
en.wikipedia.org/wiki/Printf_format_string#Type_field
I started to use perl for most of those kinda things years ago.
But indeed AWK is a usefull tool.
Beside stuff like AWK, SED or PERL differs you from the dotNot kiddies.
1% raise 😂
Thanks for the tutorial!
Here we work to live, we don't live to work. Social life is more important than work.
0:10 before I read about the emacs command I was a very productive employee. My boss would assign me a task and I would just do it.
Then I read about emacs and now my boss assigns me a task and I spent half a day configuring emacs and another half to write something in emacs that gets the job done
1.5% raise for the gentleman please.
When I look at the command you ended up with I think if it was me it would have taken me less time to write it in python that I already know than to learn all that in awk.