Thanks for mentioning ScrapeGraphAI, I'm one of the co-founders, we have implemented new features like code generator for scraping to minimize the number of calls to LLM on sites that have a shared structure on different pages, we are preparing something big related to KG, stay tuned :))))
Thanks for mentioning Crawl4Ai! I'm adding some new features, such as extracting all media tags (video, image, audio), Breadth-First Search (BFS) Crawling, and more. I do it with the aim to generate quality data without relying on large language models (LLM). I think firing up GPUs for just crawling data from a page with billions of parameters is a bit over the top. Developers can use LLMs themselves once they have the right raw data from web sources.
I just use selenium web driver and JavaScript or Jquery to interact with and get the parts of pages I want. If they use cloud flare or other bot blocking you can run js in console and utilize the copy command then paste in a txt file
For jina reader Api key free for 1 million tokens which was 570 sites then pay 10 for 500 mil worth is 250k sites which is totally insane just pay the tiny amount for much better rate limits
Thank you so much for sharing this valubale information. It is absouletly helpful. But, is it possible,as far as jina ai is concerned, to specify in the code the number of pages that I want to scrape, as spmetimes the pdf file has more than 500 pages .
Do any of these solutions work on sites you have to log in to? You can give them a url, but if the site requires you to log in, you will not be able to scrape further.
@@engineerprompt If any of these solutions are Chromium based, then one could load the page, go through the authentication process, and select the page to be scraped. Then invoke the scraping tool.
There are a couple of reasons. 1. Even if you were to just copy and paste, the you not preserve the structure in most cases, there will be table, images etc which will mess up the formatting. 2. Even if copy paste were to give you perfect results, you can scale that to 100s or 10,000s of webpages. Using these automated tools, you need to provide list of urls and they will be able to parse at scale.
If you want to build robust RAG applications based on your own datasets, this is for you: prompt-s-site.thinkific.com/courses/rag
Thanks for mentioning ScrapeGraphAI, I'm one of the co-founders, we have implemented new features like code generator for scraping to minimize the number of calls to LLM on sites that have a shared structure on different pages, we are preparing something big related to KG, stay tuned :))))
Thanks for mentioning Crawl4Ai! I'm adding some new features, such as extracting all media tags (video, image, audio), Breadth-First Search (BFS) Crawling, and more. I do it with the aim to generate quality data without relying on large language models (LLM). I think firing up GPUs for just crawling data from a page with billions of parameters is a bit over the top. Developers can use LLMs themselves once they have the right raw data from web sources.
Crawl4AI is shaping up pretty nicely. I will do a deep dive on it.
Yes PLEASE, Do a videos on {Crawl4Ai and ScrapeGraphAI}, and thank you for everything you do and your time 🙏
Yes, its on my list.
I just use selenium web driver and JavaScript or Jquery to interact with and get the parts of pages I want. If they use cloud flare or other bot blocking you can run js in console and utilize the copy command then paste in a txt file
Is there any path for learning you can recommend me? i´m generating reports from a web using python, looking for an alternative. Thanks in advance.
For jina reader Api key free for 1 million tokens which was 570 sites then pay 10 for 500 mil worth is 250k sites which is totally insane just pay the tiny amount for much better rate limits
Great review. Please do a review on ScrapeGraphAI. Maybe a comparison to Uncle Code's Crawl4AI? I like Crawl4AI and hope UC incorporates PDF options.
thanks, yes, both of them are on my TODO list.
Thank you so much for sharing this valuable information. It is absolutely helpful.
Glad it was helpful!
Super handy, thanks 🙏
Nice comparison! Please continue work on scraping for AI applications. Hot topic!
thanks, will do
Scrapegraph is pretty amazing, highly recommended
Can you make a detailed video on scrapegraphai? It’s kinda buggy right now for me
Thank you so much for sharing this valubale information. It is absouletly helpful. But, is it possible,as far as jina ai is concerned, to specify in the code the number of pages that I want to scrape, as spmetimes the pdf file has more than 500 pages .
I am not sure, their api seems to be very simple and I haven't noticed any customizations yet.
Crawl4ai sounds perfect!
Thank you. If you could dive deeper into scrapegraph, specifically the knowledge graph feature.
thanks, will look into it.
I need this materials very much,, can you share codes and api brothe??
link to the notebook is in the video description.
@@engineerprompt thanks this is crucial ...best for you dude
Do any of these solutions work on sites you have to log in to? You can give them a url, but if the site requires you to log in, you will not be able to scrape further.
Good question, I am not sure. you might have to add authentication yourself to these.
@@engineerprompt If any of these solutions are Chromium based, then one could load the page, go through the authentication process, and select the page to be scraped. Then invoke the scraping tool.
Probably a silly question, but in what is all this complicated proccess better than doing a simple copy paste from the url?
There are a couple of reasons.
1. Even if you were to just copy and paste, the you not preserve the structure in most cases, there will be table, images etc which will mess up the formatting.
2. Even if copy paste were to give you perfect results, you can scale that to 100s or 10,000s of webpages. Using these automated tools, you need to provide list of urls and they will be able to parse at scale.
Good luck copy pasting+ cleaning millions of pages for llm feeding.
Also good luck for manual updating :)
brilliant 🙌🙌
thanks :)
Thanks a lot! :)
Are there any scrapper available for LinkedIn and Instagram?
I am not aware of any.
great thx!
The android in the thumbnail looks like he's DJing. Like he's ready to drop a sick beat...NOW!
We must create order from the messiness! 😎🤖
Agree :)