PARSING EXTREMELY NESTED JSON: USING PYTHON | RECURSION
Вставка
- Опубліковано 22 лип 2024
- This is a video for those wanting to stop nightmares from 𝐧𝐞𝐬𝐭𝐞𝐝 𝐉𝐒𝐎𝐍 files. This is about as nested as you get in this video. We will use recursion to store all of the keys by index and save them as dot notation. This will keep every single level of keys and values. If you want to scare coworkers with your new found skills, watch this. Also, I left an easter egg for those who pay attention. Turn on the 🔔 notification
Join this channel to get access to perks:
/ @mrfugudatascience
➡ Patreon: / mrfugudatasci
➡ Buy Me A Coffee: www.buymeacoffee.com/mrfuguda...
➡ Github: github.com/MrFuguDataScience
➡ Twitter: @MrFuguDataSci
➡ Instagram: @mrfugudatascience
Code will be with the other 2 JSON files on Github
𝗥𝗲𝗳𝗲𝗿 𝗮 𝗙𝗿𝗶𝗲𝗻𝗱 𝗟𝗶𝗻𝗸 𝗭𝗮𝘇𝘇𝗹𝗲: refer.zazzlereferral.com/mrfu...
I will receive a small fee if you make a purchase on Zazzle of $25 or more
𝗣𝗿𝗶𝗻𝘁𝗶𝗳𝘆 𝗥𝗲𝗳𝗲𝗿𝗿𝗮𝗹 𝗢𝗳𝗳𝗲𝗿: I get a small commission if you make 3 purchases
try.printify.com/skupntonxtrn
𝐕𝐢𝐝𝐞𝐨𝐬 𝐘𝐨𝐮 𝐌𝐚𝐲 𝐀𝐥𝐬𝐨 𝐋𝐢𝐤𝐞:
▶️ CREATE NESTED (JSON) DICTIONARY: PYTHON, with pitfalls: • HOW TO CREATE NESTED J...
▶️ CONVERT NESTED JSON TO DATA FRAME WITH PYTHON CREATE FUNCTION TO STORE NESTED, UN-NESTED DATA: • HOW TO CONVERT NESTED ...
▶️ REGULAR EXPRESSIONS (Regex) for Parsing ADDRESSES using Python: • HOW TO TUTORIAL: REGUL...
Picture for thumbnail : pixabay.com/users/absolutvisi..., instagram: absolutvision
Picture in video: Tilgenerpictures on Instagram
Music in scary clip: UA-cam scary sounds
Music &. Intro Pic: Special Thanks
Pixabay: instagram (subscribe gif): @imotivationitas
Music: Oshóva - Tidal Dance on
Soundcloud: / osh-va ,
youtube: / @oshova9190
#nestedjson, #parsejson,#mrfugudatascience - Наука та технологія
Here are 2 video that you may like with JSON data:
Join this channel to get access to perks:
ua-cam.com/channels/bni-TDI-Ub8VlGaP8HLTNw.htmljoin
Code will be with the other 2 JSON files on Github
𝐀𝐦𝐚𝐳𝐨𝐧 𝐀𝐟𝐟𝐢𝐥𝐢𝐚𝐭𝐞 𝐋𝐢𝐧𝐤𝐬: (I receive a small commission on purchases)
* Prices & Availability Subject to change
--------------------------------------------
Apple AirTag: amzn.to/3dNAZHM
30 Free Trial Amazon Prime: amzn.to/3RhCKf9 (End Date: Dec 31, 2022 at 10:59 PM PST)
Prime Student 6 Month Free Trial: amzn.to/3wgMXQz (End Date: On going)
Audible Gift Membership: amzn.to/3pAfw7W (End Date: On Going)
Try Audible: amzn.to/3PETRWS (End Date: On Going)
Apple Certified Type C Charger & USB Wall Charger 20W with 2 cables: amzn.to/3dMdqPA
𝐕𝐢𝐝𝐞𝐨𝐬 𝐘𝐨𝐮 𝐌𝐚𝐲 𝐀𝐥𝐬𝐨 𝐋𝐢𝐤𝐞:
CONVERT NESTED JSON TO DATA FRAME WITH PYTHON CREATE FUNCTION TO STORE NESTED, UN-NESTED DATA: ua-cam.com/video/FVECTpahzCQ/v-deo.html
CREATE NESTED JSON DICTIONARY: PYTHON, with pitfalls: ua-cam.com/video/zhwmmjq1Nqg/v-deo.html
don't know if you are still here, really amazing video ! would you mind doing more video about how to scrape data from bad nested json/xml ? Thanks It would be great if you can talk about how to convert sparse json data into panda dataframe, thanks
@@user-vn4jw3ch8w , If you can leave a message on my channel page in the community tab that would help me remember. I appreciate you leaving feedback. Do you have any specific examples you need help with by the way?
@@MrFuguDataScience SEC latest filing (10Q 10K) reports
Holy moly! How could anyone have ever thought json was the answer?
Thanks for the comment. 😀
This particular video of reading JSON is one of the best I've ever seen. Hope it gets more views. Thanks for your efforts to explain all this.
Thank you so much for the feedback. I appreciate it.
For this Video, I am subscribing your Channel 🙏
I’m struggling with getting a json output from an api and converting it to a dataframe that preserves the hierarchy of the nested data. It’s a very large dataset with expected output of 15000 total rows. I got it done by looping over each row of parent data and json_normalizing each child row with recursion. However this takes a very long time. ChatGPT is suggesting to do this with dicts but I can’t get it to work! Note each row of data has a column called child that has the same columns as parent and can go down to four levels for each parent row.
I have a suggestion of a dictionary-list. Storing the keys and all values as a list. This will preserve the order and if you have empty entries you add some empty string or what every you need. But, you will have to flatten everything out first. If you don't want to do this, check if the data are repeating in structure and pull the first key that stores all nested information and store that repeatedly. Without seeing your data I have no idea.
This is brilliant! I've been working to flatten and parse JSON files where the nested portions are bring read as text and therefore won't flatten using traditional Pandas/JSON/third-party packages. I can't call them by index either as they're not lists and calling by element position like [1] simply returns the position of a character within the text.
I'll try applying some of the methods you're demonstrated here and see if it anything works. JSON should not be this complicated!
Thanks for the videos and all your efforts mate!
thanks for your feedback
same I was also thinking about recursion, KQL has Insane JSON objects nested, some are dictionary, some are list, some are string, and the naming conventions for entrys are NOT being followed. Some keys are "Owner", "owner", "wner", etc. Complete NIGHTMARE!!! I needed this to grab everything.
Hi Mr Fugu,
Thank you for the video, would you mind showing me the order to watch your JSON parsing videos cuz you have a few of them and I don't know where to start?
Thanks for watching. So, I made these randomly based on viewers or myself needing to solve a problem. There wasn't an order to it, kind of progressed over time to help people solve different problems. Feel free to ask questions or if you have suggestions.
@@MrFuguDataScience Thanks for the fast reply! I'll try to follow and ask if I have any questions.
hi I have used this print_dict() function it's giving me output as "['.author.lastname', 'Doe']" am not able to print index values how to overcome it. BTW appreciate your efforts 👍👍
I will need to review the video for suggestions to you. I am not sure what your data look like but I will try my best to answer soon
I would like to see your data if possible and walk through it so I can have an idea. Let's chat so I can see what is going on for you if possible
Hey Mr Fugu! Love your videos and your teaching methods!
I've been trying to flatten a rather complex json file for days now, and output it to a csv and i simply can't figure it out. My CSV file works fine, if i past it into one of the online tools, but i can't seem to be able to write the script in python.
Some advice would be very much appreciated, as i'm rather stock.
I have the JSON i can send you asap.
my laptop died a few weeks ago and im waiting for the repair. i cant help currently and havent posted for a few weeks i am sorry.
I am nowhere near this level yet but it's probably the solution I need. I'll come back in a few months lol
what do you have going on or need to solve currently?
Hi, Can you guide me on how would I load a really large json file?. I have a json file of 1.69 GB, I tried pyspark but it doesn't work, neither does pandas read_json but someone suggested me to use pickle and append lists I don't know how to do that... I searched over the internet but I don't know why I don't seem to find anything related to this... Help appreciated
Send some screen shots or code to my gmail posted on my channel page. I will look at it
@@MrFuguDataScience done.
Please show an example on read nested json data from API and stored it in DataFrame. Thank you.
Ok, good idea
also, I did do a video for getting these data from the Twitter API separately and little NLTK. ua-cam.com/video/kS_0RNdUjrQ/v-deo.html
How can I dump json data from api as json file like your twitter example?
And can you show how to dump all column in nested to preview in the same dimension with other fields in the root column?
@@jitsupayimphet3405 , Ok give me a few minutes for first question to get some code.
Second Question: I am working on that video for you, I had some hangups. But, I have a partial solution to that.
github?
It should be in decription and youtube page, but its github.com/MrFuguDataScience
this is legit
Thank you for the comment, I appreciate it.
usually watch everything on incognito. Logged in just to like this video.
thank you