Python Tutorial: Real World Example - Parsing Names From a CSV to an HTML List
Вставка
- Опубліковано 25 лип 2017
- In this Python Programming Tutorial, we'll be learning how to parse a CSV file and output the data to an HTML unordered list. This is a real world problem I ran into with my website. The list was becoming too large for me to maintain manually, so writing a Python script to automate this process saved me a lot of time in the future. Let's get started.
The code from this video can be found at:
github.com/CoreyMSchafer/code...
✅ Support My Channel Through Patreon:
/ coreyms
✅ Become a Channel Member:
/ @coreyms
✅ One-Time Contribution Through PayPal:
goo.gl/649HFY
✅ Cryptocurrency Donations:
Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot
✅ Corey's Public Amazon Wishlist
a.co/inIyro1
✅ Equipment I Use and Books I Recommend:
www.amazon.com/shop/coreyschafer
▶️ You Can Find Me On:
My Website - coreyms.com/
My Second Channel - / coreymschafer
Facebook - / coreymschafer
Twitter - / coreymschafer
Instagram - / coreymschafer
#Python
Dude you are the STAR. Please make more of those real world examples. Why don't you rather create a new category in your tutorial series "Real World Example" etc? You are just amazing Corey!
Glad you found it useful. And yes, perhaps I'll create a separate playlist of real-world examples once I get more videos like this created. Thanks!
@@coreyms will be waiting for that....
@@SanataniAryavrat Still waiting...
😆@@envy_coppr2146
@@envy_coppr2146 Waiting :p
I don't even like programming because I never understood the the core basics well. It's so cool now after your tutorials that I also try to practice. I can be the last man to indulge in programming, however, when I start watching your videos I couldn't stop watching it. Thank you so much for all these stuff
I'm gonna complete all of your Python tutorials, and then write in my resume, "Comfortable with Python - taught by Corey Schafer".
I think if in the code, Corey uses continue instead of break, it would have been better...
html_output = ''
names = []
with open('patrons.csv', 'r') as data_file:
csv_data = csv.DictReader(data_file)
# We don't want first line of bad data
next(csv_data)
for line in csv_data:
if line['FirstName'] == 'No Reward':
continue
names.append(f"{line['FirstName']} {line['LastName']}")
html_output += f'There are currently {len(names)} public contributors. Thank You!'
html_output += '
'
for name in names:
html_output += f'
\t{name}'
html_output += '
'
print(html_output)
I was almost dropping out of CS class back in 2015 when i was in second Year, then i bumped into this guy...You damn good man, you made me a Python Developer...
Corey, these videos are excellent, so clear and I appreciate your thought processes as you go through the problem. Many thanks
I love this channel! I learn more from you than from my college professor.
Very useful video for me who does a lot of manual stuff using CSV files. Keep up the good work Corey
Very nice explanation with no time-loss in between, thank you for that!
One small thing for improving your efficiency is to use str.join() method instead of += (concatenation) because it's much more slower and uses more memory.
That is because strings are immutable, hence everytime a new string needs to be created by parsing all the characters from the previous one.
Would be very interesting to see some OOD tutorials in the future since you have such an amazing and clear way of explaining stuff!
Thanks a lot for these videos Corey 😃 I just successfully automated the process of copying and arranging photos and videos in specific date folders from my dslr to my PC 😃
Every time, thank you very much. Best python channel across youtube
You have done a great service to so many people with these videos, thank you
see mine too. The playlist provides most of the fundamentals for Python. And soruce files too.
Hi Corey, you have such a fluency in python and your teaching style is really good. I appreciate it and will contribute to your efforts once i am financially capable.
Yes, we enjoy these tutorials. Thank you very much.
loved all videos and subscribed, your explanation was best than others who say "we can learn it later, what that function does!"
this was so cool! Using just two modules, CSV and Random, we can actually take in data online and generate our very own huge chunks of dump data sets for various testing and research purposes or even just for fun, to play with it :p
Thanks for this, really amazing!
Another absolutely awesome video. Thank you, Corey.
Great video Mr.Corey!! , Very useful in daily tasks.Greetings from Mexico.
Best python teach ever. Thank you corey
awesome tutorial.
thanks a lot! this makes the concept clear.
and yes, do make a video on real time use of oops in python... that would be very helpful.
As usual, great work, Corey!
- Jacob
Great video, just what I was looking for.
It's funny you posted this. I had to use this Python library recently with some data I gathered from an experiment of mine. I ended up using pandas, though. It's a nice option for data analysis/visualization and it offers functions to deal with csv files, converting them to data frames.
Thank you Corey! Your videos have been an amazing help for me. Is it possible to take each line of a csv and turn it into a separate instance of a class and assign the instance variables? My goal is to create methods and functions with the data from the csv file. Thanks again and keep up the good work!
I am new to Python, your videos are awesome
I just discovered this channel and all I can say is that you do a great job at explaining different Python concept. Could you do a video on async/await (asnycio)? And explain what asynchronous programming really is, since so many people have problems with understanding the concept.
Great lesson thanks Corey!
I like to thank you,+Coreyms for your awesome videos. Your videos are not only informative but also interesting. Can you make some videos on python GUI relate?
Thankyou Corey for all of your contribution, they help me alot on learning python. on this vid, downloading offline file and upload to your website, i'm thinking is there a more fancy way such as getting the data from Patreon API if it have?
Thanks Corey!🙏
Thanks sir 😀 , most helpful 🙂 , heart ❤️ for you
OMG you are the best seriously
First of all, your teaching style is good, and anyone grabs ur teachings easily, and thanks for teaching us. you are the best tutor I have ever come up with in python language. can u suggest any python language book, which is best to go through other than your channel? Moreover, I am a beginner to python language I thoroughly enjoyed ur videos, if u can suggest me best python book to ur sensibility and I guess it suits me and it helps me to learn further (I like physical like books). In case if u written any books related to this language can u suggest one (or) your favorite book, please......................................
Thank you Corey!
Great video, thank you!
what video do I watch that takes the html_output and insert it into the actual html page?
Excellent tutorials. Admire your style of coding and clean and neat explanations. +1 for more of these real world examples.
I seriously can not thank you enough for the amount of help you are providing. This tutorial was really helpful. One thing i'm having trouble with is practicing OOP. I've watched your tutorials on OOP and they are pretty good but i don't know where to use them and how to implement them into my daily routine of practicing python as most of the code i practice is usually in functions or sometimes not in functions at all. I hope you understood what i asked as my english is not that good. Great stuff !
Hey there. Your English seems great to me. I wish I could speak a second language as fluent as that. And yes, applying Object-Oriented concepts to everyday problems is a common hurdle that many people face. I would say that the most important part of initially understanding Object-Oriented code is so that you can look at libraries and frameworks written by other people and understand what it is they are doing. Once you start to see how others are using object oriented code in their projects, then it will slowly start to give you ideas for how you can do the same. I would love to put together some videos on practical real-world examples of object-oriented code in the future. I'll see what I can come up with. Thanks!
Thank you for the great advice ! :)
Corey Schafer
Hey that'd be great thanks
extraordinary bro
Amazing and very informative videos....
Really great and intuitive video. I just have a doubt.. what if we want the headers to be there but don't want that second line (i.e., "reward +1.....), without using the DictReader? How do I do that?
Corey thanks so much for this real life problem example.
Could you please do real life problems with pandas?
Thanks
Great tutorial !
This video was helpful, I wanted to ask if you could make a video for making a report using python, like extraction of the the data and making report .
do you have a video on how to automate that data to automatically update an Html page based off a time frame or schedule. That would be extemely helpful content for most everything I would want to do, if included with Json data for micro controller data logging. Thanks for the great videos, I really love your approach and thinking/teaching style. Thanks Corey for taking the time
Could you make a tutorial on Tkinter? (maybe you have made one already but i cant find it)
Awesome video!
Perfect example ❤ thank you Corey!!!!
Corey, thank you so much. This is great. I do have a question if you could prepare something about this. I have 2 csv files. The first column is device serial number say from 1 to 2000. Then, the rest of the columns are data like, one of the columns is jitter, one is frequency ppm, one is current etc. I want to write a script that takes file1 and file2 (same file structure, but serial number column in file2 is a subset of serial number column of file1). The script should plot the difference between elements of specified measurement column. I want to make sure these these differences will correspond to the same corresponding serial numbers of file1 and file2. Thank you.
Will you pls tell me how to do validation on accessed emails from csv file ??
Hi Corey, truly fantastic video as always.
Have you thought about making a series about some mini-midsize projects that a freshman as myself and the other viewers can take as a step towards making a portfolio for a future job interviews? To transfer your experience about project life cycle, process some examples... There are some examples around, but with your gift for lecturing, it would be one of the best I'm sure.
That's a great idea. I would love to do that. I do have some larger project tutorials on the back-burner right now. I just need to find the time to get them written and recorded. I try to keep a continuous flow of videos coming out, and when I get stuck on big projects for too long (like my OOP series or Intro to Python series), I start to hear from a lot of people who want me to push out videos faster. So I juggle between working on the large-scale projects and these shorter videos to keep them coming. But yes, I will work on something like that in the future. May take some time though.
That's understandable, you must be overloaded with wishlists :). BTW, OOP and PY intro series are a major success so just keep up the good work and we'll be waiting.
Is there a way to do somethine like "next(csv_data, occurrences=2)" , so we can skip multiple lines and keep the code a bit more clean? (min 7:02)
From Nairobi-Kenya-Africa I have come to love your work Corey. Thanks alot. Could you please do some Python GUI stuff ...
Wish you would get that app that shows your keystrokes would love to learn the hot keys you use
kudos mate
Thank you
This is great, next step, how can you get this on a HTML page?
Great tutorial
Instead of using DictReader isn't it easier to use reader and make the code also readable with a codeline like fname, lname = line[0], line[1] ?
Thank you 🙏🙏🙏
Very useful✌✌✌✌
Hey, corey , I only discovered your channel a week ago and I watched 3 videos of which I can't get enough of, as I am new to programming your vids were so helpful and full of knowledge not shown in any beginners courses, so thanks. One thing in this vid I tried but is not working well is for loop inside the with statement is running forever please take a look and tell me what is wrong:
import csv
html_output = ""
names = []
with open("prayertimes.csv","r",encoding="utf-8") as data :
csv_data = csv.reader(data)
for line in csv_data :
print(line)
Hey there... I'm not sure why this loop wouldn't end. Just by glancing at it, it looks fine to me. How many lines are there in your csv file?
hey I found the problem it's not in the script it's in another script that I made the csv file out of it's basically a web scraping script I fixed it, but thanks for the help.
That is a hard CSV file to parse. Have not seen the whole video. I am looking forward to see how you parse it.
If someone put a comma in the name, for example, and we don't want to split on that, how does the csv module handle it?
Your videos are excellent. Please consider using a larger font - it's difficult to read your code. Thank you very much.
I want to be a data scientist and am following your videos to make my base strong.Falling in love with you.....I wish I could get a mentor like you in real life....Love from India...
please do the video on parsing reading and writing csv files also.
Thanks Corey, That great little tutorial. What will it take to write the output_html to a file?
It would be pretty simple. You would have to include whatever additional HTML you want for the entire file, and then after everything is appended to the output the way you like, you would just need to do something like:
with open('webpage.html', 'w') as html_file:
html_file.write(html_output)
Thanks. Work like a charm!
What if I wanted to print the names after "No Rewards"?
Thanks you so much
thanks bro
Cool video as usual. What about a csv file that is not ordered or structured with hearder but instead a continuous string that separates data with a ; delimiter. Iterating over the data returns one really long string line. Objective is to parse the string with carriage return or new line after each ;. example ( ST*834*0001*005010X220A1;BGN*00*1*20120106*010510****4;REF*38*170175;) Must regular expressions be used in combination with a csv reader or csv writer dictreader or dictwriter?
How do we extract this exact csv file you used? Is there any link?
sir when i use f''{line[0]} {line[1]}" and after doing all according to you i get error of EOL while scanning string literal i search on net all time but i didn't found any answer please help me in this problem,thanks
Great video. But what about the names after line 35 or after 'No Reward', those are not being added to the list ?
Think you meant to use "continue" and not "break" there
remember those people didn't want to be mentioned thats why corey used break and not continue
Old comment but instead of break you should use continue at 10:23 so the "No Reward" line should skip and get the other names after maggie, iam sure you ment to change it but it just passed. iam learing a lot from your videos hopefull ill watch all of them. God bless you!
it is not allowing me to use 'break' after if statement.
this was great, how to show the result on your website automatically , do you have to upload python file to your server or how it work ?
No, I just have this script on my personal machine and occasionally run it from there. I have a login to my website where I can drop in that section of HTML
thanks Coreny , can you make video on that if you have time .
I'm not gonna lie...it was rough going, but I had chatGPT on standby to help me. LOL! The issue was weird and it may be more VS Code than anything else as when I was messing with stuff per chatGPT's suggestings, my terminal finally showed data instead of just [ ]. After that things were fine...until the f-string line where I used single quotes when I needed to use both. :)
_No Reward_ Smith contributed 1000$ !
Haha. I'll just have to hope no one chooses that as their first name. If so, the script will need to be altered a bit.
lol
Erellei - It could be Google or Anaconda contributing.
you need to make videos on python mini and advanced projects
As I'm not currently on Python 3.6 yet, how would you append to names without using the f-string?
I've tried pretty much anything from the "String Formatting - Advanced Operations for Dicts, Lists, Numbers, and Dates" video, but none of it gives a satisfactory result. What am I doing wrong? Python version is 3.5.2.
The format method should work for you. I'm not sure why the examples in that video wouldn't work. I opened my python interpreter and created an empty list and appended a name to it twice with these two different methods:
>>> names = []
>>> line = {'FirstName': 'John', 'LastName': 'Smith'}
>>> names.append('{} {}'.format(line['FirstName'], line['LastName']))
>>> # OR
>>> names.append('{0[FirstName]} {0[LastName]}'.format(line))
>>> names
['John Smith', 'John Smith']
>>> # You could also do this
>>> names.append('{FirstName} {LastName}'.format(**line))
So you have several options... I hope one of these work for you. If you'd like explanations of any of those then I would try watching the string formatting video through one more time.
In my last version I had the format(line) *outside* of the names.append method, and I had "line" instead of 0.
I will rewatch the video to try and understand why the 0 works and 'line' does not...
I need to also get my head around this unpacking business... 8-)
Thank you very much for attempting with me.
to test the code in the same time with youw need patrons.csv
5:41 you are able to print the csv_data when parsing it through list(), when I do this i get an "I/O operation on a closed file error". python v 3.6 - what am I doing wrong? Using anaconda(IDE)
Are you sure your indentation is correct? It sounds like you may not have the line indented correctly and you are operating on the file object outside of the context manager
It even happened when I copied and pasted your source code in! must be something wrong with my builds?? - huge props for responding btw I know how busy you must be as your channel is so popular.
I tried it in visual studio and it worked - must be something wrong with my Anaconda install - I will reinstall it.
Which Python IDE u r using ?
I personally just use the pandas module for any IO python related tasks these days.
You should do some HTML tutorials
Can we have videos on webscraping please
I didn’t understand the term of html_output += "......."??
love your videos corey!! but you talk sooo fast, haha as a non native speaker and programming beginner i always watch you at 0.75 speed.
using keyword 'continue' instead of the 'break', this can be a better understanding of 'continue'.
you can add this code:
data = pd.read_csv('file1.csv', error_bad_lines=False, warn_bad_lines=False)
Hey Corey ... little help what if someone just wants to skip row 2 and row 35 only ?
Hmm, that's a pretty unique use case. In that case I would probably just initiate a counter outside of the loop and then increment by 1 each time through the loop. Put in an "if" statement that skips the loops using "continue" if it's the lines you want to skip.
yeah i tried to do same but it didn't help me when there's some random text line at row 9 text was 'amount...', at 23 it's 'shopping...' and similarly, at some other row number it's 'generate...', i was looking for an algorithm that can identify the rows and differentiate such rows which do not make sense like in your case it's first name, last name and email and rest of the data is irrelevant, I can't read the data or would not give the effort to identify such rows because in my case there are over 1 million rows......
Saadeep, All you have to do is to replace break with continue in your code. for the data file used in this case. However, for the case your are trying to solve I will suggest to try with regular expression.
Need python 3.10/3.9
Decompile parse
Such an awesome real-world example. Thanks!
I just used this info to do my first 'real' project that makes a list of Steam games by player count (based on achievements) since I had that CSV file handy. Can't even imagine how long it would take to enter 13281 * 2 table data cells by hand!
kevinblanzy.com/secret/steam_game_list.html
i get error in this lines. names.append(f"{line[0]} {line[1]}")
video time at 9:01 to 9:14
Error:list index out of range.
i am using python version 3.6.6
In your code also i get error in same line.
thanks for this tutorial.i really appreciate your work!!
Hi Parth, if you can share your code on Repl it or Paste Bin, for example, I can try to help you... =D
Double check the csv file and see if there is a blank line at the end. There should NOT be one. That's what was wrong with mine when I did this.
did u manage to solve ? I have the same error
import csv
html_output=''
names=[]
data_file=open('TechCrunchcontinentalUSA.csv')
csv_data= csv.reader(data_file,delimiter=',')
print(csv_data)
after this...its showing this output Why? the file have 3000+ lines
[Finished in 0.2s]
can we use pandas package
Sure
10:32 Why cut out the rests of the names after Maggie Jefferson? Aren't they important? 🤔🤔
This is the right code, for line in csv_data:
if line[0] != 'No Reward':
names.append(f'{line[0]} {line[1]}')
you could order name
C - Corey
S - Schafer
V - is Vegetarian
couldn't find a word for V 😜
¯\_(ツ)_/¯