You have the absolute best videos on UA-cam!!! I'm resisting the urge to type in all caps right now lol but seriously, this video just helped me finish a $200 project!! Thanks again for all you do for the community 🙏🙏🙏
After seeing many videos and trials to do web scraping on secured websites, this has finally brought the solution. Thank you so much! Attention on cookies: Playwright acts as a new/clean browser. So opening a website from the script is like visiting it for the first time. I discovered that the website I wanted to scrape, started with a cookie banner that you have to click. So before filling in the username and password, I had to do a page.click('button#btn-accept-cookies')
Great video John, as usual! I started using playwirght a few months ago and prefer it to selenium or helium, it is much faster, way less error prone and it is being updated constantly.
Hi John. Just wanted to say thank you and please keep making these videos. I have been studying Data Analytics online and just got a job offer for analytics position. Even thought it does not directly require programming skills your helped me to stay motivated, opened up opportunities for automation and inspired to do some interesting projects. Thanks again and keep it up!
I've been religiously watching your videos for the last week or so. Such a great source of information, you're a great teacher, very direct and to the point! I've succesfully set up a project scraping data from betting sites to find arbitrage opportunities - mainly via hidden API's. But some pesky websites seem to restrict their APIs - hoping to solve this with playwright :).
Mate you're the best for this stuff. Your deadpan style also makes me laugh. I bet you have a wicked sense of humour. Remind me of the russians. Dry as anything, and wizards with code!
Thanks for the tutorial... I think the demo site has changed though, the last part of the script does not work. In particular the html output of page.inner_html('#content') looks nothing like the demo and the subsequent steps do not return the results in the tutoral.
thanks. unforuntately this is often the case, things change- this is why i try to demo the methods rather than specific sites. but it just furthers the need for me to build my own web scraping test site!
@@JohnWatsonRooney I have the same issue. Although it doesn't make sense because I can see the h2 tags in the html enclosed in online. It seems like playwright is ignoring the h2 tags. When I print(html) after the line html = page.inner_html('#content'), the result in the editor does not show any h2 tags. It doesn't come close to the section of code I see online.
I sure do. If I need to run a headless browser I use playwright the other tools like httpx and selectolax do different things and are my go to for making requests and parsing html
10 out of10 again! Haven't installed Playwright yet and wondered how you found it for speed vs Selenium? In an earlier reply you mention that you prefer PyCharm now over VS Code. Will the community version work for most or do we need the Pro version?
Hey John, I was precisely looking up for a technique like this for an upcoming project I'm aligned to where we need to login inside one of our company's internal web tool and scrape the leads generation table that appears post loggin in, write it to an excel file and resulting file will be attached to a Bi dashboard for automatic updates and publishing. Will this technique of yours work or would you care to give some more of yours experts advice? Thanks for being there. As a self taught pyhtoneer new to programming, you give me a lot of hope with your content. Thanks for being there for people like me. ❤❤❤❤
Thanks, Koushig. I have a question: When I log into google I get the following message: "This browser or app may not be secure" error when trying to sign in with Google on desktop apps
I'm trying to follow along with Spyder, but I'm getting this error. What's wrong? Error: It looks like you are using Playwright Sync API inside the asyncio loop. Please use the Async API instead.
I'm wondering how we could scrape multiple pages, I've watched the crawl and follow links with scrapy video but I don't know if FormRequest is the way to go instead of playwright.
Thanks! Depends what I am trying to achieve. If it’s something like this I wouldn’t bother with Scrapy. One of the videos coming up with be scrapy + playwright
@@JohnWatsonRooney cool! Looking forward to that! I digged through your videos and got a bit confused with Itemloader. Should i use it if I just have to get just very static job info? I don't really need to process the data
@@vt2788 if your project is working for you without it then no don;t worry. it does make is easier when adding to databse etc as you can use it to clean your data out properly and stucture it
Hii..Mr. John, I'm working on a playwright Python project where I want to print the response.json() of a particular response. Kindly make a video on the request-response in the playwright.
Great video mate! really helpful! Question - any idea how page content can be displayed while using pytest with bs4? my tests passing successfully but i can't "scrape" data from websites so i can't see all the information in the inner_html. I'm using vscode both as IDE and terminal and besides passed tests in terminal, there is no other information. any ideas?
Thanks for your great video. I have 2 problems can you help me about it: 1. Use playwright to crawl website. But after click on button --> ajax call --> how i can reload data from ajax response. 2. After use playwright to login, can we use scrapy to send new request and crawl data.
Hi John. When I got the the line, sync_playwright, I got the error, saying that I was using the sync api inside the asyncio loop. Do you know how to resolve it?
Guys, please help me! Let's take a simple scenario, open a browser, go to google, search for 'word', press search, and the scripts ends. In selenium, after the search, the browser is still open and usable, i can browse through the search results. In playwright, the browser closes, even if i did not us browser.close(). How can I keep my browser open and analyze the search results of my google query?
Hey! It’s being run in the context manager which automatically closes the browser when the code is finished. In the docs there is a bit about running it without the context manager this is what you want
@@MancePax no problem, look here: playwright.dev/python/docs/intro and go to the section "Interactive mode (REPL)". this code will work in your code editor too and you should be able to take it from there
I am logging to a site with 2F authentication. First there is a captcha and then after keying captcha there is an OTP. How do we code to accept user input of captcha and OTP (selenium or playwright)? Help will.be appreciated
I have faced this error while applying the code, any suggestion? Looks like Playwright was just installed or updated. ║ ║ Please run the following command to download new browsers
This video is also awesome. Thanks for sharing your knowledge with us. But I got the following error. Can you please help me for solving the error?
File "D:\Project\My_Py\untitled2.py", line 10, in with sync_playwright() as p: File "C:\Users\user\Anaconda3\lib\site-packages\playwright\sync_api\_context_manager.py", line 45, in __enter__ raise Error( Error: It looks like you are using Playwright Sync API inside the asyncio loop. Please use the Async API instead.
Traceback (most recent call last): File "c:\Users\Sellitrage\Desktop\playwright test1\main.py", line 1, in from playwright.sync_api import sync_playwright File "C:\Users\Sellitrage\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\sync_api\__init__.py", line 25, in facing this error while running dont know how to solve this....please guide me.
This is the first time I got introduced to Playwright, so much more elegant than Selenium. Thanks for sharing!
Great content man, keep it up. One of the best channels on scraping and automation
You have the absolute best videos on UA-cam!!! I'm resisting the urge to type in all caps right now lol but seriously, this video just helped me finish a $200 project!! Thanks again for all you do for the community 🙏🙏🙏
After seeing many videos and trials to do web scraping on secured websites, this has finally brought the solution. Thank you so much!
Attention on cookies: Playwright acts as a new/clean browser. So opening a website from the script is like visiting it for the first time. I discovered that the website I wanted to scrape, started with a cookie banner that you have to click. So before filling in the username and password, I had to do a page.click('button#btn-accept-cookies')
Thanks! Good tip too thank you for sharing
Great video John, as usual! I started using playwirght a few months ago and prefer it to selenium or helium, it is much faster, way less error prone and it is being updated constantly.
Yes totally agree
Hi John. Just wanted to say thank you and please keep making these videos. I have been studying Data Analytics online and just got a job offer for analytics position. Even thought it does not directly require programming skills your helped me to stay motivated, opened up opportunities for automation and inspired to do some interesting projects.
Thanks again and keep it up!
Thank you, very kind! I’m glad you have enjoyed my videos
no nonsense and to the point short video. thanks John
Thanks for watching!
Great Work Man!
Always right to the point and no fillers. Thanks for you hard work 🙌!
Thanks!
Thanks! How to retrieve the html using Playwright and pass it to Bs4 was exactly what I needed to know. Great video.
Great stuff, as usual. I am using stuff I learn from your channel almost everyday.
Great video and practical example! Would like to see more advanced stuff with playwright.
Very helpful as usual. I look forward to your Playwright series
Thank you for a useful and concise tutorial. So many Playwright videos are all tied up with Pytest, which I don't need for my usecases.
My first intro into Playwright :)
It's awesome, thank you!
Glad you like it!
Getting Selenium running on Google Cloud Platform was so much stress.
Last night I setup Playwright with no stress, no extra work was needed.
That’s great!
I've been religiously watching your videos for the last week or so.
Such a great source of information, you're a great teacher, very direct and to the point!
I've succesfully set up a project scraping data from betting sites to find arbitrage opportunities - mainly via hidden API's.
But some pesky websites seem to restrict their APIs - hoping to solve this with playwright :).
Thanks a lot I appreciate it! Good luck with your projects!
Wow! I absolutely love your videos! New subscriber coming in 🎉
It's my luck to find your video,it make me learn python much easer.THANKS!
That’s great I’m glad you found it!
I like this, very useful, and it can be combined with Selenium if desired...
Excellent tute John, thank you for such clarity
2:38
bro thought we couldnt see "bye"???
btw your video helped greatly!
Very informative video.
Looking for more videos on playwright.
More to come!
Thanks for sharing these awesome tutorials. You sir, are a gem
Glad you like them!
Wow, this is much cleaner than selenium
thanks john useful as usual
Glad you enjoyed it
Thank you John, please make more videos of playwright.
Yes more coming!
Mate you're the best for this stuff. Your deadpan style also makes me laugh. I bet you have a wicked sense of humour. Remind me of the russians. Dry as anything, and wizards with code!
Thanks! I try not to take myself too seriously!
It look much easier and cleaner than selenium. Thanks for the great video.
Thanks, that’s why I like it more
Thanks for sharing john, this is a great tutorial ever
You are very welcome
Thank you very much. You made things so much simple, easy Thanks a lot
Good competition between Cypress vs Playwright vs Selenium.
Thanks for this video. Love from Pakistan 🇵🇰
Thank you my brother!
Thank you so much. It helped me a lot..
Dropping by to say thank you for this tutorial!!!! And also ask which theme did you use?
Thanks- it’s one of the GitHub themes, my favourite for vs code at the moment
Thanks John for sharing. How can I reuse the login session?
Great video!!....is there more tutorials/videos on playwright (for scraping)?
Thanks! Master Rooney.
Great work 👌👏💯
Finally my Selenium webdriver headache has gone :D
Any advice on getting Chromium driver to work? Webkit will work ;-)
An afternoon, cookie session, soup, json, all pushed to Google sheets webapp.. Loving this!
subscribed....... please keep on making videos like that ...thanks
Thank you!
Thank you for sharing, i like your channel. Keep it up mate.
i got trouble, RuntimeError: Event loop is closed, when i ran the code in 7:28p. what are these errors?
Thanks for the tutorial... I think the demo site has changed though, the last part of the script does not work. In particular the html output of page.inner_html('#content') looks nothing like the demo and the subsequent steps do not return the results in the tutoral.
thanks. unforuntately this is often the case, things change- this is why i try to demo the methods rather than specific sites. but it just furthers the need for me to build my own web scraping test site!
@@JohnWatsonRooney I have the same issue. Although it doesn't make sense because I can see the h2 tags in the html enclosed in online. It seems like playwright is ignoring the h2 tags. When I print(html) after the line html = page.inner_html('#content'), the result in the editor does not show any h2 tags. It doesn't come close to the section of code I see online.
Realy help me thanks a lot i need to hide browser page after connect in for loop
Hi John, I would like to ask if you still use Playwrite? or do you have new favorite? Like the tools I learn from you httpx and selectolax.
I sure do. If I need to run a headless browser I use playwright the other tools like httpx and selectolax do different things and are my go to for making requests and parsing html
@@JohnWatsonRooney thank you very much for responding. :)
very interesting, thank you
Can playwright be pointed to the browser installed on the machine instead of the one that is playwright installs?
Yes you can connect via cdp(?) protocol to a running browser - it’s in the docs somewhere I’m sure
10 out of10 again!
Haven't installed Playwright yet and wondered how you found it for speed vs Selenium?
In an earlier reply you mention that you prefer PyCharm now over VS Code. Will the community version work for most or do we need the Pro version?
I think it’s about the same speed wise I just find it much easier to work with - yes PyCharm community is good it’s what I use!
hey great video i loved it just wondering if it's possible to open multiple tabs simultaneously ?
Hey John,
I was precisely looking up for a technique like this for an upcoming project I'm aligned to where we need to login inside one of our company's internal web tool and scrape the leads generation table that appears post loggin in, write it to an excel file and resulting file will be attached to a Bi dashboard for automatic updates and publishing.
Will this technique of yours work or would you care to give some more of yours experts advice?
Thanks for being there.
As a self taught pyhtoneer new to programming, you give me a lot of hope with your content.
Thanks for being there for people like me. ❤❤❤❤
Great video! Thanks
You're welcome!
Hi, does Playwright or its browser(s) need to be updated when a newer version of the browser that we are using for the scraping gets released?
what code editor do you use?
This is VS Code with one of the Github Themes - Dark Dimmed I believe. Honestly I prefer PyCharm nowadays
Beautiful!!! 🙂
John, no words to thank you for all your tutorials! I hope you will show us more cases based on playwright. Thank you!!!
Thanks a lot!
As always, a great video John! As an expert in Requests, do you know if it is possible to use Playwright together with Requests? Tks
thanks bro, do you have any idea how to bypass a captcha with playwright ?
you resolved ?
Hello, please tell me, is it possible to somehow get the har file of a browser page without browsermob proxy?
i write the exact same code but it doesnt seems to be working.can you help me out,John?
i am facing huge gap of loading a secong page after a first page with this method and still nothing is printing in console, please help what to do
Thanks, Koushig. I have a question: When I log into google I get the following message: "This browser or app may not be secure" error when trying to sign in with Google on desktop apps
wow how do you run headed browser just like that? on my wsl it wants me to xvfb-run (whatever this is) for some reason
I think it works differently in WSL - if you did it on windows or Linux it would work just like this
@@JohnWatsonRooney i must try that then, thx
Thanks 😁
AWESOME!!!! 🙌🙌🙌
I'm trying to follow along with Spyder, but I'm getting this error. What's wrong?
Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.
Well done John, thank you so much . How fast is it Vs selenium?
it's still got to open and run a browser but so far I think its faster
Bro how keeping browser open to collect datas dynamics?
Using jupyter notebook, I get: "It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead."
Hey so I don’t think you can use this with jupyter due to the way the internal loop is used, you’d have to write your own script instead in a py file
@@JohnWatsonRooney Thanks and thanks for your video! I've read the same thing online, I'll try and see if it works in the a py file
which theme are you running on vs code?
It’s one of the GitHub official themes, i quite like it
Amazing content Thanks sir
Nice video. Is there a java equivalent for this? I also would like to have a word or two with you
Playwright works with Java - playwright.dev/java/
You can email me if you like my email is on my main YT page.
@@JohnWatsonRooney Thank you John. I emailed but I am yet to get a response from you. Just thought to reach out and let you know
Hi, John can you make a video about asynchronous playwright to scrape multiple URLs?
I'm wondering how we could scrape multiple pages, I've watched the crawl and follow links with scrapy video but I don't know if FormRequest is the way to go instead of playwright.
total_orders = soup.find('h2', {'class': 'pull-right'}).text
AttributeError: 'NoneType' object has no attribute 'text'
what's the reason of this ?
@@abel4776 same here, but after removing the 'text' I got as a result: "total orders = None". The code is exactly the same as show in the video.
Hello John, please can playwright scrap Angular web pages?
Yes it should easily be able too - angular is a JavaScript framework so playwright is a good option.
Great! How do you decide whether to use playwright+bs or scrapy?
Thanks! Depends what I am trying to achieve. If it’s something like this I wouldn’t bother with Scrapy. One of the videos coming up with be scrapy + playwright
@@JohnWatsonRooney cool! Looking forward to that! I digged through your videos and got a bit confused with Itemloader. Should i use it if I just have to get just very static job info? I don't really need to process the data
@@vt2788 if your project is working for you without it then no don;t worry. it does make is easier when adding to databse etc as you can use it to clean your data out properly and stucture it
@@JohnWatsonRooneyBrilliant! Okay, I see how I could customize my scraping that way. Thanks so much!
Hii..Mr. John,
I'm working on a playwright Python project where I want to print the response.json() of a particular response. Kindly make a video on the request-response in the playwright.
Great video mate! really helpful!
Question - any idea how page content can be displayed while using pytest with bs4? my tests passing successfully but i can't "scrape" data from websites so i can't see all the information in the inner_html. I'm using vscode both as IDE and terminal and besides passed tests in terminal, there is no other information. any ideas?
Thanks for your great video. I have 2 problems can you help me about it:
1. Use playwright to crawl website. But after click on button --> ajax call --> how i can reload data from ajax response.
2. After use playwright to login, can we use scrapy to send new request and crawl data.
Hi John. When I got the the line, sync_playwright, I got the error, saying that I was using the sync api inside the asyncio loop. Do you know how to resolve it?
That and the AttributeError: “PlaywrightContextManager” object has no attribute “_playwright”
Great video! BTW, can it log in website with recaptcha?
No, recaptcha is not supported by selenium
Guys, please help me! Let's take a simple scenario, open a browser, go to google, search for 'word', press search, and the scripts ends. In selenium, after the search, the browser is still open and usable, i can browse through the search results. In playwright, the browser closes, even if i did not us browser.close(). How can I keep my browser open and analyze the search results of my google query?
Hey! It’s being run in the context manager which automatically closes the browser when the code is finished. In the docs there is a bit about running it without the context manager this is what you want
@@JohnWatsonRooney I have no idea what you mean :) but I will dig around!
@@MancePax no problem, look here: playwright.dev/python/docs/intro and go to the section "Interactive mode (REPL)". this code will work in your code editor too and you should be able to take it from there
Which theme you use?
That one is one of the GitHub themes
@@JohnWatsonRooney okay thanks. it's really good theme
thank you so much
I am logging to a site with 2F authentication. First there is a captcha and then after keying captcha there is an OTP. How do we code to accept user input of captcha and OTP (selenium or playwright)?
Help will.be appreciated
Using code in video I get this error, how do I fix, thanks.
page = browser.new_page()
^
SyntaxError: invalid syntax
What about captcha?
Nice!
you are the magician boss :)
Unfortunately, this project is not working purposes... The sites I need to log in to say these browsers are outdated... :(
Really? That’s odd, try reinstalling playwright it should fetch the latest version of chrome
Nice... How unique with helium? Seems bit lazy code ;) compared your tutorial. I'll try my end.
I keep getting No module named 'playwright'
John you are a great teacher. Thanks for your efforts. Can you please make a video for Amazon and Playwright. you know Black days are coming. Thanks
Thanks for sharing this video I was wondering can you please make a short maybe an hour long python crash course
Thanks a lot
I’ve thought about it but I wasn’t sure as there are lots of good videos out there already like that. You’ve given me some good idea though
I have faced this error while applying the code, any suggestion?
Looks like Playwright was just installed or updated. ║
║ Please run the following command to download new browsers
I ger error RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'Page.goto' was never awaited
Can you help me?
Nice Video
wow! cooooool
This video is also awesome. Thanks for sharing your knowledge with us. But I got the following error. Can you please help me for solving the error?
File "D:\Project\My_Py\untitled2.py", line 10, in
with sync_playwright() as p:
File "C:\Users\user\Anaconda3\lib\site-packages\playwright\sync_api\_context_manager.py", line 45, in __enter__
raise Error(
Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.
Hmm are you running this in a Jupiter notebook or similar?
Traceback (most recent call last):
File "c:\Users\Sellitrage\Desktop\playwright test1\main.py", line 1, in
from playwright.sync_api import sync_playwright
File "C:\Users\Sellitrage\AppData\Local\Programs\Python\Python310\lib\site-packages\playwright\sync_api\__init__.py", line 25, in
facing this error while running dont know how to solve this....please guide me.