What Makes A Good Cli Tool | Prime Explains
Вставка
- Опубліковано 24 сер 2023
- Recorded live on twitch, GET IN
/ theprimeagen
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/deeznuts - Наука та технологія
It's weird how many developers don't even know how to use the command line.
It's nice to be among developers who don't know how to use them, because I can basically be a magician who gets information out of seemingly nowhere in an instant. To those who know, I just used grep.
it's forgivable coming from a junior dev, but I facepalm in shock when I see it coming from someone who has been in the industry for 20+ years
we have been spoiled with fancy graphics and it only gets more and more unfamiliar. plus no one really teaches it.
@@ardnys35 I think some people also find it daunting since CLI tools are quite volatile. They do exactly what you tell them to do without questioning anything. That's also why they are great (besides being fast) but you can mess things up if you do the wrong thing.
Then again, people seem to be less afraid of SQL despite being able to nuke an entire table by forgetting to add WHERE.
I die a little inside every time my manager says "that's nice, but does it have a GUI?"
Well that was an epic nugget of knowledge. Can we get more of that please?
Agreed!
Please do the frontend masters course for writing good unixoid CLI tools! Including golden shower tests!
I want to put in my vote for the Frontend Masters course on this. Thanks!
@3:25 You can accomplish the same thing with shell redirections, or *gasp* the dreaded cat piping.
All tools should start with stdin and stdout. If you really wanna build in file handling fine, but you don't have to.
Yeah. Moreover, when I heard him say that using files functionality support is cool for integration tests I was wondering "well, but you are not testing stdin input that way". And with pipe redirection you actually can.
Never underestimating the power of piping.
But specifying stdin to be a file can be useful. On cmd or powershell 1.0, piping removes all non-ascii characters.
It's called tee because it makes a T. Data flows across the top of the T (from stdin to stdout), but also gets duplicated out the bottom (to a file). Super useful, and not just for :w !sudo tee %
wait wha's the point of :w !sudo tee % ?
Save a root-protected file if you opened vim as a normal user.
very good explanation actually
Wait, :w can write into the stdin of a shell command?
@@theodorealenas3171 It can! It can even do it for range writes (i.e. writing from visual line mode), if you only want to send a segment.
Thanks to you I recently came to enjoy gnu parallels to parallelise some build scripts.
This is really useful as someone who's recently working on CLI tools.
The Unix philosophy is like functional programming.
It's an apt description, indeed of the Unix philosophy.
If I'd categorize it, I'd file it under 'Dataflow programming' but that's a more or less orthogonal concept.
the issue isn't the pipelines, it's all the basic commands you have to learn. takes time to remember them all (the basic ones).
but once you know em then yeah it's pretty much functional programming
@@bitti1975 I think you're misunderstanding / misusing the term orthogonal. Because "dataflow programming" is most certainly a core tenet of FP. Clear input and output. Referential transparency and all. These are not unrelated. But I guess you could say it's a simplified description of FP, instead.
That being said, I don't think anybody had FP in mind when the Unix Philosophy was "created" as it were.
Yep.
UNIX's shell redirects and pipes can be derived formally as an I/O monad.
It's not a matter of opinion: it's exactly, formally, provably like functional programming
Would love a course on tooling!
very helpful tips. thank you prime!
this is the kind of advice I love mr prime for.
Thanks for sharing this. Imagine reading this tip as an article when you can watch a video instead... oh man
This casual 4 and a half minute vod has more useful information than hours long tutorials
What do I need to do to end up working on stuff like this?
I'm really bored at my current job (make one webapp finish next, do it super fast so the project is cheap)
PLEASE do a course on writing dev tools! 🙏🙏🙏
I believe the right way to call tee(1) is `tee hee`.
"Command-line Tools can be 235x Faster than your Hadoop Cluster" -Adam Drake
The good old Golden Shower Test
gnu parallel is used a lot in bioinformatics or other scientific computing areas, they make a script in python or perl and then gnu parallel.
Your best video so far ⚒️ thanks
Please do that course :)
He didn’t said “-agen” 😭😭😭
please do a tips n tricks for common cli tools like grep
all great developers understand how unix works
Where to learn more about those things???
POSIX compliance is a must.
Watching you alone brings me a whole level of inner joy!😂
We need a "last cli class you'll ever need"
I'm glad this became a UA-cam video-AGEN
C++ has two stream outputing to stderr "std::cerr" and "std::clog". The latter one really makes it clear that this is where the logs should go to (and not to "std::cout").
tee has been my secret sauce since forever
I'm a little confused on the inclusion of optional file reading/writing - would it not be simpler to only support stdio at the program level, then use angle brackets or cat/tee for if you want files? To me it seems that would also follow "do one thing and do it well", although with how often I see the pattern it seems more likely I'm just missing something here
speed/throughput
just go and test the difference of:
cat some_file | grep some_regex
vs
grep some_regex some_file
@@kuhluhOG that's a useless use of cat - of course it's less performant than redirections. I actually see slightly improved performance with redirections instead of file argument, although it's the difference between averaging 1.21s and 1.23s (maybe zsh caching, maybe some reduced syscalls, don't really know why)
@@ShadowKestrel point still stands. Maybe you want to run something multiple times on some compressed data, you could use lz4 -d -c | debug-tool or you could decompress to a file first and then save decompression overhead, piping IO overhead (which is very significant), and you enable some really powerful optimizations such as memory mapping or pre-allocation based on file size etc.
Because redirection is a function of the shell and not all usage of the tool is within the shell.
It's useful for a few reasons. Off the top of my head, you can specify multiple input files at once, you give explicit options for anyone using your tool in a script, and the tool would likely behave better when not being launched from a full shell environment.
Please do the course. Would be very happy
good advice
Do you have a sample repo to look at?
Please can we normalise man pages with CLI tools - I honestly don't think a tool should reach v1 without a man page. I don't want to figure out what your help flags are and have a ton of scollback maybe printed to stdout or maybe to stderr, I just want to write `man ` and use my favourite $PAGER to browse the options, example usage, etc.
Please do the course!!!
My comment is,
I cannot believe prime made this a youtube video.
You said Golden Shower, I google that, it was nothing about software testing. 🥺
Amazing diagram!
What is that drawing app he is using?
I'm always torn on the file escapes thing. Because it does kinda feel like it violates the UNIX philosophy to integrate that into my program when I could always replace "foo --input ./bar --output ./baz" with "cat ./bar | foo > ./baz", but that does also make it kind of a pain to use on files
Cat is for concatinating not for reading single files.
foo < ./bar > ./baz
@@joshix833 No one ever bothered to teach me the
@@Howtheheckarehandleswitoof. I felt that one, I started using it after a year of using Linux, or more
but where do status updates fit into this?
re: tee... yes! or at least write a script which writes data out if not already stored. Too often people try to consume and process data in the same loop and it puts unnecessary strain on the data source and complicates debugging. Just write it out somewhere and use that.
I wouldn't call it a "break point" but I understand what is meant by it. It's literally a T... like a pipe T... 1 input and 2 outputs of the same data. The syscall does the same.
hey i work as a web dev with 2 years experience. but i feel, im so lack about how actually that thing work or how to make something other than a web server or web client. maybe because i started learning programming trying to build a website.
primeagen video appear on my youtube home, and i think he is what we called an engineer or a dev not just 'react dev' or etc.
What step should i go to learning this kind of thing ? i mean like memory, gc, and many more i hope you know what i mean
I hope anyone here who have lot of experience as 'software engineer' can tell me what should i learn first and the learning path.
It all comes down to understanding the fundamentals, understanding c, assembly, memory management, compilers and operating systems, even some hardware and cpu architectures, can give you insights about how your browser and JS engine works (both implemented in c++ utilizing many fundamental concepts mentioned above) that the absolute majority of ‘exclusively web devs’ will never try to begin to understand. It can enhance the quality of your code, broaden the things you can do and make it easier and faster for you to learn new technologies. Those who took a CS degree had to go through a few semesters of learning the fundamentals. Prime has a CS degree so he had to go through that too. However a CS degree is absolutely not mandatory to get a deep understanding of fundamentals since everything exists online
@@abcdefg-nu4xj hey, thanks for the answer !
I think you are right, i lack everything about computer scince since i learn programming not from cs degree
I search on youtube and harvard make a video about CS with 24h long. I hope it can help me to learn some computer science.
Thank you for your answer 😄
@@gabumon9210 i would suggest taking it one step at a time, dedicate a few days to learning c well, focusing on pointers and memory. Then maybe try and implement a few data structures (dynamic array, hashtable, red-black tree) and algorithms from scratch with c, then you can go a little lower level and learn assembly , focus on how your C code translates to assembly, this is a great chance to learn about CPU architecture and get a deeper knowledge about memory. The next thing from here is to learn about operating systems, namely linux. First learn how to interact with the OS as a “user” with syscalls to spawn threads, manage memory and IO. Then, with your knowledge of c, data structures and sssembly you can go deeper and learn how central parts of the OS are implemented, how it manages threads, I/O , memory, etc.
Each step might take you a few weeks or months to get right but once you’re done you will have a strong foundation to learn about things that build on those fundamentals like networking, compilers, virtual machines, web browsers, etc
Anything that interests you
Your learning sources should be yt videos, free (or paid) courses and books
a good cli tool can easily integrate into a pipeline.
damn, i knew about tee but never thought of it as a "pipe-breakpoint" I always have to refetch a big pipeline 😔
To learn this, fire up a unix machine from, uh 35 years ago and do some work.
Those ideas are utter gold.
And have short options. '-v' is a hell of a lot quicker on a 2400 baud VT100 than '--version'
(But I am ancient)
please do the 'good tools' course... I would pay big dollars for you to do it!!!!
make a course on this homie
Lithium batteries, interchangable heads, pulsing and vibration functions, ....Oh sorry I misread the video title .
what kind of football play is that? 0:22
Someone send this man a weebo tablet for these excalidraw moments 😂
@primetime do a video showing this in code - that would help people understand
A good CLI tool also must return a signal. Returning a good defined integer is awesome . Sometimes you don't give a 5hit about strings. ( Windows programmers nowadays have no clue about this.)
ok, i'm too junior to understand, need some code examples. anyone konws a github repo that shows what he said?
lessssgoooooooooo!
Teez Nutz
Inversed whiteboard
Jźzz ... programs read stdin/file and output to stdout/file and stderr.
That's what the professor told is in 1982 teaching Pascal and Fortran simultaneously
Why put in all the effort for file inputs and outputs? Isn't that what redirects are for? And like, sure, redirects don't support multiple files for standard in, but you can just use cat for that, no?
Most of the time yes. But e.g. grep needs file arguments to be able to show source files and lines.
if you only care about using a single file, check the args and freopen the standard streams when appropriate. then you get to pretend like you're only using the standard streams, but you're not inconveniencing your users
for example, with xargs, you can have a list of files to operate on, one-per line, and say "
Sometimes you do need to seek though
@@joshix833 Sure, but that's an exception; that's not a reason to bake it into most tools
@@pineberryfox I suppose it's slightly more convenient with xargs but can't you also just do a cat with xargs? But fair enough
I use CLI utilities like there's no believing
pipe is the only way to pipe
Windows Terminal ftw
If only there was a language that outlined CLI commands like this drawing...
bash/zsh/csh/fish do not count in this regard for those wondering
"what is golden shower?" (Bolsonaro, 2020)
I want to write a better cli calculator but i know i shouldn't
Personally I use python for calculations in the cmd just the standard repl/IPython
Most languages (php, ruby, python) or frameworks (node, laravel artisan) have interactive shells, just input an expression and voila calculation complete
@@user-kc8uh9ve4j I use the python repl too, I just have the craving to make something with a better interface. auto implied last result, proper juxtaposition multiply, lists and automatic mapping of scalar functions, cli option to output a cleaned version of last session, etc
Which is why I definitely shouldn't, I don't think I actually have that kind of time
Depends on your metrics for good but qalc is pretty awesome for a semi reliable DWIM calculator
@@billeterk That does look pretty darn good, thanks
you look a gta character
I feel like I'm missing 90% of the context here.
File in/out is just duplicating stdin/out
potato
Just dropping a comment to be the first.
git push faster way of doing things into brain
MERGE CONFLICT
You are only talking, code once ıf you can
🥴
those are most of his videos. he doesn't really do programming and if he does, he's constantly talking to the chat. its a different type of channel and it isn't for everyone.
parallel sucks I use xargs
this is wrong on so many levels. when processing a file, the stdin and stdout IS the file stream. there's no separate channel for files.
Real talk, how many developers watching this video who have a job didn't know and use parallel and tee at least once already (not necessarily together)? As a Linux guy I know my command line like the back of my hand, but not every developer is a Linux guy. I know 2 other developers personally and one of them is really not familiar with it at all and the other knows enough to get the job done but hates using it, or so he tells me. I'm curious what the demographic here is like.