This CLI Tool is AMAZING | Prime Reacts
Вставка
- Опубліковано 27 чер 2024
- Recorded live on twitch, GET IN
/ theprimeagen
Article: alexplescan.com/posts/2023/08...
Author: Alex Plescan | alexplescan.com/
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/deeznuts - Наука та технологія
Former programmer and current linux/unix administrator I've beaten many of our in house Java/Python(many multi-threaded/processor) routines that massage data with parallel + Awk (or Mawk for even faster performace, given you fit the limitations). We crunch through massive amounts of plain text / hour. Nearly every time the younger staff are perplexed at how this can be (which is in turn perplexing to me how they think they will beat a nearly 40 year old highly optimized program with a few days of coding). Sometimes Mawk will beat my C++ implementations compiled with the most aggressive flags I can set.
I hear so many people mention AWK feels like i should spend a weekend with AWK and try it out.
@@BlackwaterEl1te If you need to manipulate text, CSV, or basically anything textual nothing beats the ease of use being able to isolate columns, rows, line numbers, regular expressions, etc. Having some of the worlds best computer scientists optimize its code doesn't hurt either.
@@CommanderRiker0 Cool thanks for the info, AWK sounds really useful especially in this text data heavy world.
@@BlackwaterEl1te I recommend the book "Sed & AWK" (Make sure its a later edition), the free pdf is floating around everywhere on the net (O'reilly publisher). Just skip the SED parts if you want. SED is a line editor for quick and dirty regex replacements (its basically built into VIM). AWK is a full programming language.
Lol u have chatgpt now, awk is easier then ever
I use a lot of bash scripts not because I like bash scripts, literally any other scripting language would be better, but if written properly, it just works on almost on anything... Also GNU parallel is awesome.
I work with docker images and sometimes even bash is too much to ask for. alpine falls back to sh
More like works on almost nothing. Aside from built-ins, every command is an external program that may or may not be on the system. And the program may have different versions or implementations. Sometimes it's GNU tar, sometimes it's BSD tar. To make a platform independent bash script you need to find the common denominator of every target platform.
Don't get me wrong, I use bash all the time, but it's because it's actually a very good model for composing programs together.
the ::: _is_ parallel specific nothing to do with bash
parallel is also written in perl
Yea the ::: are just good ol' arguments to parallel
It's indeed written in Perl. Hence the stupid :::
> be primeagen
> wax poetic about the benefits of gnu parallel
> never run it on stream
Por que Maria?
Parallels is great for anything you could do using multiple tmux sessions. Anything that requires joining and error handling gets overly complicated pretty quickly.
It’s written in Perl.
Author of GNU parallel also has a UA-cam playlist references in the manual: ua-cam.com/play/PL284C9FF2488BC6D1.html&si=lTOiasQ-llm9IwEB
I feel like there are so many amazing tools out there that you would never find in a google search if you search for a specific problem. Isn't there like a comprehensive list of those?
A note on xargs: in POSIX it has no parallelism. I don't know how much that means though.
That's good to know. I'm glad I use parallel for all my parallelism :)
Also big fan
@@ThePrimeTimeagen you're a blessing. I gathered Reddit people now and "percentage viewed"s dropped.
"$?" only stores the return code of last executed command
I wish I saw this live so I could spam "man parallel" in the chat; ::: is specific to GNU parallel
same here, it was just too frustrating to see all those comments getting ignored
@@hiraginoyuki Eh it wasn't frustrating imo, it was fun seeing Papa Prime try to figure out whether it was yet another weird BASH construct or not
I had the same question about the ::: when I started seeing examples of the parallel command and did the same thing went crazy digging around perl, and bash and zsh and fish trying to figure out what this thing was and couldn't find anything but it is part of parallel its how it separates arguments or you can use -arg-file
I surprised prime didn't know that I had to find out as well and found it in his book "GNU Parallel 2018 ,Ole Tange" section 2.1 'Input sources' it's the first thing he talks about. No, I wonder when it was introduced... dam I'll have to do some more digging.
Parallel is written in Perl >
6:28 'brace expansion', if you want to look it up
Acadamia has knowledge about GNU parallel for a long time. This is what happens with big tech is so focussed on their own stack. They ignore the amazing open source contributions made by smarter people who know how to simplify things.
Academia is about that money. Most of the tools used in college are used as advertisment. Most students end up with jobs that used those classroom commercial tools. I had to learn about parallel via perl in my sparetime and night job; I was my university's web administrator and saw it in a script that some guy in the 1990s wrote. Thank god, I stumbled upon the old timers' scripts.
@@complexity5545 Wasn't my experience. I got to use whatever I wanted.
And so did my professor.
I'm sure your point is valid in certain circumstances.
Parallel is awesome, I even used it a few times where I didn't even need parallelization because it's argument parsing and hand handling is so much more powerful than xargs so I just used it instead
parallel is written in perl
i searched a few hours ago about gnu parallel and prime comes out with a vid, what are the odds Hmm Susge
It's like hearing about a new tech thing and that day seeing a Fireship "100 seconds of the thing"
The odds are about 7/77
Mmm sausage.
Parallel written in Perl :)
The most glorious 15,000 lines of Perl ever written.
Finally!!! Thank yooouu
You're learning Spanish, I'd like to see how many native Spanish speakers we are ... greetings from the Dominican Republic.
Do you want to count us with the likes? Or you want a reply from each one of us? 😂 😂
There's quite a few from what I know. Greetings from Argentina
Greetings, also from Argentina. Love el primo agen
Dominican too
This is kind of surprising that he didn't know ::: is cartesian product (in parallel). Its basically data set math/stat like designs to create a argument list of commands. I n bash cartesian product is like `echo {a..b}{1..2}` and it'll print a1 a2 b1 b2. Its SQL like understanding too. Its used to reduce invocations and source code (especially in maths).
He probably knew it though, but just forgot (or only saw it used a handful of times by the solo smart old-timers).
GNU Parallel and Tmux are the two tools I rely on daily for my work. Indispensable.
Watched for the headline. Stayed for your narration.
yeah, he should have asked chat-gippity for the turbo fish thing :D
This could solve a lot of problem at my job, we have a bunch of e2e tests that are crazy, some developer that left try build a multilprocess version with node but it suck and it fail for no reason
this episode was extra amusing as I am currently writing a bash script to auto play Cookie Clicker using only core linux/X11 tools, not just a clicker but full on buying,gardening, combo'ing etc
12 minutes later and I still don’t know what it actually does…
im going to write an assembler in bash. you cant stop me.
Man, both `:::` and `{1}` are totally a GNU Parallel syntax.
Indeed, as zsh on MacOS says:
❯ ls {1} ::: {1..10}
ls: 1: No such file or directory
ls: 10: No such file or directory
ls: 2: No such file or directory
ls: 3: No such file or directory
ls: 4: No such file or directory
ls: 5: No such file or directory
ls: 6: No such file or directory
ls: 7: No such file or directory
ls: 8: No such file or directory
ls: 9: No such file or directory
ls: :::: No such file or directory
ls: {1}: No such file or directory
so... it seems that parallel is written in ... perl
read the doc, it's awesome
The name is Themultitaskgen
Parallel is a superpower
yeah, and when after 100+ attempts you finally get all the quoting/escaping/piping just right in your interactive bash session, you can begin fixing quoting/escaping/piping required for a script version
1080p is the new 4k!
Have you ever heard of VIM though?
Luke Smith sold me my first hit of GNU parallel.
As someone who has programmed altogether too many lines of Bash (all 10 of them were 10 too many), I know I've done a little too much Bashing (excellent name for it, BTW. It's called bashing because that's what I do to the keyboard with my head) when I see any control flow statements more complex than an "if". "While" is right out, and "for" gives me PTSD (dear God, may I never try to implement flags in Bash). Also, anything more than 5 pipes or escape characters on one line are heinous to read, and I've colorized my command prompt. Why does double quote string expansion escape single quotes, I'll never know.
If you saw my zls colors regex for my completions you would trip out😂😂😂
The name is the parallelagen. XD
Dude, Primeagen, you need to take a break and slow down sometimes. Take it easy and enjoy it, there's no rush!
dbrock losing his damn mind in the chat. love it
Again, a suggestion for another mindless Rust-rewrite?😳
Ghanoo Parallel
11:57 I'm serious
Wow, it's like Powershell on Windows ten years ago.
Powershell got something right? I thought everybody hated it.
Parallel was made in like 2002, back when SMP was being written in kernels, and google started promoting python instead of perl. During that time most programs were written to only run on 1 cpu. Parallel allowed you to take a 1 cpu program and use all 4 of your CPUs. Today we have 128-Cores and parallel will apply to all of them; stuff gets done quick. We'd use it in clusters and then Gearman was made. I think Powershell copied alot of that stuff.
bash or pass
At least the article didn’t call anything mesg
found this on SO: Parallel processing makes sense when your work is CPU bound (the CPU does the work, and the peripherals are mostly idle) but here, you are trying to improve the performance of a task which is I/O bound (the CPU is mostly idle, waiting for a busy peripheral). In this situation, adding parallelism will only add congestion, as multiple tasks will be fighting over the already-starved I/O bandwidth between them.
On macOS, the system already indexes all your data anyway (including the contents of word-processing documents, PDFs, email messages, etc); there's a friendly magnifying glass on the menu bar at the upper right where you can access a much faster and more versatile search, called Spotlight. (Though I agree that some of the more sophisticated controls of find are missing; and the "user friendly" design gets in the way for me when it guesses what I want, and guesses wrong.)
Yeeeah finally this shit it out :)
BTW: love your content ^^ I watch it all some days ;)
Greetings from germany 🇩🇪
yeah C baby !!!
Anyone remembers which is the video on this channel where @ThePrimeTime spoke about gnu parallel being able to continue to the next command down the pipe without blocking?
There were also some diagrams drawn in that video.
Found it! thanks to youtube's good search algo, searching "parallel" led to ua-cam.com/video/pHJmmTivG1k/v-deo.html
Stop paralyzing start parallelizing
I want to date the girl from algoexpert, is the only ad that I don't hate opposed to clement's mental picture.
😂😂👍👍
Go sleep parmesan
My eyes burn every time I read bash, pwsh and cmd. The technology is still not advanced enough for a comprehensible command line language.
Nushell and Fish have readable scripts.
pwsh more readable then cmd or bash to me, still not great
Or is it really gnu/Linux parallel
better to skip parallel and just use ansible
agree, he is probably Junior dev
Ansible to run tests? Wtf?
@@greyshopleskin2315 xD you probably didn't saw more then 5k test suites in your life in automated CI/CD lol
@@ordinarygg no, but I don’t understand the use case for ansible.
I used it to configure machines, that’s it.
How do you use it?
You have a cluster of n machines, and use ansible to pull latests changes from git repo and run tests across all machines? Or what?
@@greyshopleskin2315 pull from repo? This is security breach to give access machine to run tests to your repo, it should be synced outside.
In general yes, but you missed:
- collect the results
- bootstrap env to run those tests if they are E2E
- create a report
If you use something like gitlab CI or even worse github actions imagine it will go down or change the syntax LOL.
All kids that use free things don't understand one simple rule, if it's free then price is you and your code. Nothing is free in this world.
If you know Python forget about this tool and just use Multiprocessing module -> you will have queue, shared memory and more. Bash is very limited and hard to read in terms of just fundamental programming logic.
Do you use it to invoke other programs?
@@complexity5545 read about multiprocessing, shell and async Pipes, you can open programs and async stream
What's the point of this tool? Basically all modern scripting languages can do the same thing with the same amount of code. Except you aren't using a shell language anymore = Win.
gnu is not unix
Your bash bashing is uncalled-for. Adhere your own advice and learn your tools!
yes, i agree, but as i said, once i hit arrays, its time to use something better
i still think this is a good piece of advice for bash
@@ThePrimeTimeagen Yes, there should be a threshold, and in the examples given using an array was unnecessary (that's why we have globs, e.g. `for test in potentially_flaky_*.sh; do...`). Still, there is also a threshold for switching to a 'real' scripting language, since interacting with CLI tools is so much easier in bash than in python, ruby and alike. And using an advanced feature can keep the script more readable if used appropriately.
Btw. brace expansion (à la project/{src,dist}) I don't consider an 'advanced feature', since it's also very practical in interactive use (i.e. in particular when you don't write a script). And it's not even bash specific and supported by practically all shells (fish doesn't have range expansions though).
4:36
why not just use a makefile for this? feels like it would work well enough with the parallelization, but now it's more understandable, and I don't need to download a new utility.
(maybe something like uhhhh)
```make
# tests
T=potentially_flaky_1.sh potentially_flaky_2.sh potentially_flaky_3.sh potentially_flaky_4.sh potentially_flaky_5.sh
tests: $T
potentially_flaky_%.sh:
bash $@
```
and then just exec the tests with
```sh
make -j tests
```
and you can still get output in the correct order with just a couple tweaks to the makefile:
```make
# output files
O=pft_out_0 pft_out_1 pft_out_2 pft_out_3 pft_out_4 pft_out_5
tests: $O
cat $O ; rm $O
pft_out_%: potentially_flaky_%.sh
> $^ ; bash $^ 1>$@ 2>&1
```
love. nuf sed