The BEST CLI Tool
Вставка
- Опубліковано 4 лют 2025
- Recorded live on twitch, GET IN
/ theprimeagen
Become a backend engineer. Its my favorite site
boot.dev/?prom...
This is also the best way to support me is to support yourself becoming a better backend engineer.
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-K...
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/dee...
I've been using jq for years.
I do a lot of troubleshooting as part of my jobs, which means making a lot of ad-hoc API calls to JSON APIs and comparing them to SOT.
JQ is so beautiful for wrangling API responses. Processing large JSON files. You can calculate stuff, reshape stuff, feed it to other stuff. Absolutely beautiful
I thought only i knew about this XD
Stamps:
1:35 - Reading a file
1:39 - Compact (-c)
1:45 - Specific field (.)
1:50 - Raw (-r)
2:21 - Showing keys (keys)
2:31 - New object
3:00 - Select not null fields
3:31 - has()
3:47 - Math
4:59 - Contains
Literally yesterday I was making plans for my first project using json, and I thought, "There HAS to be some CLI tool out there to make this easy. I'll look into that later." Thanks for saving me the trouble!
Look up one more character and search for jqp. Soooo nice. :)
If only I'd have known this a year ago. And it's been there on my computer for who knows how long! Very useful - thanks for sharing.
I used this to deal with a 9GB json file I was handed.
Split it into 90MB parts before my database ingest and it saved me so much effort.
Do you want the Earth to collapse into a black hole under the weight of 9 gigbyte JSON? Are you nut?
I love jq. I've been doing a lot of work recently with ElasticSearch and honestly jq is required kit for ES work. Combine jq with bog-standard bash pipelines and you can do some serious magic.
two other text manipulating commands that are useful are 'awk' and 'sed'
This is actually pretty sick, I wish I would have known about this literally 2 hours ago where I had a big json file I wanted to get some info from
for me was las week... fumbled like a mf...
I use yq (the YAML equivalent) in my deploy jobs for a lightweight way of updating argocd applications in my IAC repos. Works great.
My god. This left me speechless, I know NOTHING about CLI tools! Thank you Prime for showing us this!
Everything useful was first a CLI tool. Nothing you can’t do.
@@TheStickofWar It always comes back to command line and config files.
“what could this guy possibly so hyped about-oh yeah jq is really cool and i should remember it more often”
Still learning and only understand 15% of what you talk about but you’re still entertaining and motivate me to grind, that’s talent.
Your wife's idea of a good tool, is different
nice nice nice nice ligma nice nice
what's ligma
ligma balls @@darukutsu
@@wowvain2989 GOTTEM
@@wowvain2989 lmao
There's nothing more satisfying than a well crafted jq
For those of you who are okay with non-posix shells, nushell is basically a whole shell built for this type of workflow and with all types of data. Its pretty pretty fun imo
This! NuShell is truly fantastic, it's a shell built around the idea of structured data like this, except EVERY built in command (like ls) also outputs structured data to be transformed. Plus built in parsers for CSV, JSON, Excel and many more. Scripting is also just wonderful, with a Rust- inspired language.
I tried out nushell some time ago and I feel like its got the same problem that powershell suffers from, namely that dealing with objects is just as often an annoyance than it is useful.
nushell definitely isn't as bloated as pwsh nor requires you to sell your soul to microsoft, but for that reason I could never use it as my main shell.
Definitely a good tool for data querying/manipulation though.
This sounds like my next tool time moment. 👍
Update: Awesome it's in the Termux repos 😸
Was about to comment the same. I haven't looked into jq much since I can do the same and more with Nushell without being limited to just json
I am waiting for nushell to mature and stabilize, its awesome but changes too often and scripting just recently became a thing
Well, congratulations Primeagen for being of one of the lucky 10.000s recently
Wait what, this is the first time your hearing about this tool, just wait theres `yq` for yml
The new golang yq is the bomb diggety. Old python yq was whack
There's also xq, but I prefer xmllint --xpath as it's pre-installed on most systems
@@replikvltyoutube3727 It's called xslt, and it's been around for 25 years. xsltproc is the cli tool.
The yq I have installed includes xq and jq. It makes shell scripting suck just a little less.
YQ is a wrapper over JQ 😂
It always warms my heart seeing devs learn more about the myriad of cli tooling available. Just wait till you see parallel
It's been on some vid but just not featured as the main topic. He seemed to appreciate it.
Wait until Primeagen discovers "jc", a dark brother of "jq", then it completely changes the interaction with the output of any core linux/bsd utility.
If you want to fiddle with the queries without having to type the same command again and again, there's a tool called jqp that lets you query json (and yaml) files and preview the results, and whenever you're satisfied you can save the query to clipboard, all without leaving the terminal.
Man finally learns a useful programming language and is blown away.
It's great to see jq to be maintained again. Great tool that deserves to be maintained.
jq came in clutch today at work. Messy ass mongodb doo doo nightmare and it got everything filtered down by an obscure regex to a nice human readable form for troubleshooting. Thanks for the tip!
I once played with jq and suddenly I had test reporting system, where from every test (pretty complex tests covering automated system upgrades) I collected result in JSON, aggregated them into one big JSON file and then outside the pipeline I had a processor which could turn each of these JSON's into a series of static HTML files (one for each subset based on domain-specific filter).
The whole processor thing was built entirely in jq, using its library system. It was.. great times. Seriously. Sure, the system had its limits but working with jq as a language was fun and since jq is a functional language I actually learned a lot by having to shift the paradigm.
When I use jq, sometimes it's really nice to use the crate "fltn" to pipe json to fzf, and then maybe even pipe my interactive selection back into jq.
In my course lab we need to use jq for querying json from a web api. I remember watching this video 3 weeks ago and ran over to revise things. Literally a godsend
THANK YOU! I have been looking for something like this for the past 2-3 months to form a "lookup" tool against our ...struggling... database that houses non-sensitive document titles and tags; this will make life 100000% easier than figuring out per-user database queries!
Nice one. You could consider making short videos for such cli tools. It's great motivation for using them
That's the kind of content I started to love you back in the days. Awesome
Prime pushed me get more knowledgable at my cli
one of the 1st things after neovim was jq, surprise that you did not know this earlier.
Im actually using to pipe from a curl and into a xargs to perform queries in a row
Pretty slick. I usually end up jamming together some array ops in JS in the browser console when needing to quickly get info out of a bunch of json objects.
I knew and I used jq for quite a long time, but instead I didn't knew about I can pipe a vim buffer to different program xD
9:20 "Tech humor: jq json jokes"
Definitely great for parsing curl responses that give you a json token. I randomly came across this when trying to parse out an oauth token so I could stick it in a header for api testing
Been using this bad boy for a week and oh my God I've been missing out badly. More quick tutorials like these and how they work for you!!!
I've been using nvim wrong too that !jq there opened my eyes
this video is going to change my life in more ways than prime can imagine.
I'm happy you finally found that great tool. I've been playing with it since 2020. I do even more than that. I also edit json files with it. I even flatten json files with it (not that it always works) but those stuff are great, yeah.
Im searching for that CLI thing all my life! Thank you!
i made something similar for web scraping at one point but other than that i have no idea what you need this for so i want to ask what you do with it
So this is what it actually feels to see Primeagen talk about something that you know what he's talking about?
After watching this video I decided to try jq. I had a pcap file that I need to process 10K packets. I exported the packets as json and was able to do a bunch of data reduction on the command line. I found it to be easier than using a Jupyter notebook with Pandas. Thanks for the info.
0:55 that’s a Quagmire pause during flight announcement.
Actual tool is crazy.
i understand and agree with your excitement it's a God sent jewel to make my bash scripts less depressing
jq is life, it makes the AWS cli tool actually usable
Hahah yeah been there before , the aws cli is a bit shit
Used jq with aws cli for years, got super comfortable with it but then took an azure gig and now their garbage json query language has broken my understanding of the syntax for both.
That opening was like poetry. jq is a vital tool.
I have to use jq so infrequently, I always forget the query syntax and have to look it up again.
That's my use-case also, and I don't use it all that infrequently, so I tend to use jet instead of jq, but that only sidesteps the problem if you use clojure on a regular basis.
okay, so I wanted to find an interactive version of jq, and the names I came across in that rabbit hole ended up being quite hilarious
there's jid, which allows interactive json filtering
than there's jiq, which also allows interactive json filtering, but uses jq as its backend
then there's jaq, which tries to be jq, but fixing some of the things that jq does wrong
we get to jnv, which allows interactive filtering of json files; they started by using jq as their backend, but now they use jaq as their backend
oh, also jql exists, which is yet another json query language tool, but that does not try to be like jq, but which did inspire the author of jaq to make jaq, though jaq does try to be like jq, unlike jql
4:54, you dont have to attack me like that . That personal bruh...
it goes very well with jless!
This is why I love the comments section. Now I have a new tool to pair with jq and jet.
This seems way better than my current tactic of opening a browser inspector with the console open and using a bunch of manipulation with JS to spit out what I want. Going to download it now, thanks!
jq is awesome. Yet there's more power if you dive into the nushell. It's like a shell with jq and rust-like typed scripting built-in
Before starting this video I thought what tool he could have discovered and nushell was the only thing I could think of to be excited about CLI-wise in the past few years.
Sick bro squeal for jezson 😂 🎉🎉
Every time I get overly excited about some tool, I look at my wife with this look on my face that says 'I really need to tell you about how cool this tool is I found' and she Immediately knows I'm about to nerd out... 😂
For those who don't know, then there are substitutes for yaml and xml also: yq and xq
*notification for new prime video*
Speaking of getting excited for a tool
I use powershell to do this. Can not only read/select/filter json but also xml and csv
That’s cool. How does this work using PowerShell?
@@tom_marsdenPowerShell turns JSON into an in-memory object. The shell offers other commands for querying, mapping, and selecting any collection.
Everything mentioned in this video is just standard PowerShell operation. All of the techniques you use to query JSON you could use for other structured data types or .NET objects.
@@tom_marsdenpiping to convertfrom-json is the way. It just spits out a native powershell object/hashtable.
When I read the title, I explicitly thought of JQ.
I have never heard of Jake Weary till today... when I watched the retelling of "It follows". And that very hour Prime mentions him. Damn, these coinncidences are to stop me being atheist one day
I use jq too infrequently to remember the syntax despite using it for years.
using cli to visualise data in 2024... still... I didn't think we'll be here 10 years ago.
This video didnt just teach me more jq than I already have been using, you taught me more about (neo)vim as well
LEGEND
You should also check yq (same as jq but for YAML) and tomnomnom/gron (it flattens the json keys so it's easier to find a key hierarchy and to grep the data).
Programmers when they reinvent the wheel again:
4:43 - 4:58 Poor arch users. Prime went too harsh on them. 😅😅
good call getting this off the clips channel, jq needs more love than that
You just pitched JQ to your wifey?! 😂 A legend ⭐
This video was like "wait there's more"
I just always use jq for parsing out data from AWS secrets manager in a bash script... Used it for years but nice to see it can do way more.
fz I find for my json digging sessions even more useful.
Mixing it with fzf makes even more awesome
I use jq and yq every so often when I need it. However, because I use it infrequently, I need to go back and learn it again and again
jq's the bomb, I've been using it for years - it's super helpful for processing kubectl -ojson output. As others have noted, yq is equally great when you need to process yaml...
Might have to check that out. Usually I just paste json into a browser console and use javascript. Handy to see it in a text editor though.
My jq-fu is so weak - all these things that you could do with it makes me feel like a toddler trying to fit the square block through the round hole.
Never knew that Jquery had such an awesome command line interface!
The hair on prime while selected text
hell yeah. best kept secret ever
Prime would probably loose his mind if looking at Nushell ❤ JSON, YAML, CSV, TOML, SQL... all at once!
knew about jq...but always struggled with the syntax. thx homie
I love how you've got "jQuery jokes" on your chatgpt history haha
I knew jq for a longtime just never played around with it this much
OMG, I wish I knew about this before. Nice tool
You might wanna look at the article "let futures be futures"
This is absolutely insane
btw, there's also yq if you happen to have a whole bunch of yaml that you want to parse through with the same syntax.
i like this jay-sawn query.
Jq in terminal is awesome, but jq with vim is incredible!! Installing now.... thanks Prime!!!
~ This video is 101% useful!! ~
@theprimetime could you please describe what the hell were you doing before finding this out?
were you like creating a new js app to filter things every time you needed to walk through data?
or were you grepping and hopping?
I use dataweave which works for many things like XML CSV JSON, the only down side is it is not open source and I don’t think it is actively maintained too
You can even pipe a curl to a rest api into jq that returns json.. awesome! Great for testing an api from the terminal.. thanks for sharing! If I could add jq to a bash script that’s part of a cron, you could do some interesting data extraction and save it to a new json file to be evaluated or transferred into a folder that an application actually can read or display through a web interface.
Nice 😊! If I wanna look for logs in a humongous log file with a certain timestamp for debugging, this is super super useful! Thanks 😊 prime! NGL this looks super familiar though, I may have used something similar, but this is neat - gonna try it! Yee haw 🤠! - Amy
This is a cool tool, I didn't know it existed. I usually just write some basic python code to parse json files. But this seems much faster and easier.
I've been building a JS library similar to this CLI for a while to add to my repo as first project.
Supports most things SQL does: Fields, Expressions, Aggregates, DomainAggregates, filtering, sorting. JOINS not yet
Not sure if its worth finishing now as people might see it as reinventing the wheel.
Should I continue working on it ?
nothing wrong with reinventing the wheel if you learn while doing it
That's pretty slick...you can do pretty much all of that with PowerShell (not sure about the vim filter, but the rest), but it's still pretty cool and I'll probably mess around with it later
Honestly very surprised to see that so many devs don't know jq. If you've tried to work with any sort of web API in a shell script, jq is like the tool they teach in 101.
Jq was my best friend last year, last month, last week, yesterday, today as well
Cool, using it often, btw did you know there's also a yq (same thing for yaml)
I actually hate the way jq is a whole new language and is pretty much essential today with how widespread (and useful) JSON is.
I actually started using powershell on mac because its support for objects and converting from json is so much better.
I wish I had it yesterday when I had to remove some playlists from a massive Spotify backup json file.
didnt know Ninja switched careers
What the fucking hell bro. I've been using jq only for formatting my shitty api response and never knew all of that was possible. That's just fkn amazing bro!! Definitely giving it a try!