To jump to the next page you will need to add the logic. For the example, the agent will only go to the URL provided and return the dataset for the page.
Are you able to handle complex sites such as Expedia (you have to go past pop-up and “load more” elements) using swarm? Also how about bot detection? Nature websites have anti bot measures in place
In cases where LLM rate limits are low, will this significantly affect the procedure? I am accessing 4o-mini from a third party service, and their rate limit is 5 requests per minute.
It's a great resource to get started! Thanks for making that kind of stuff. However, can you please make another video by using Ollama, as it is free to use. I hope you will try..❤
Does this will scrap even sites that has pages? Like click the button to pass and get the next content?
You will have to implement additional logic to address the pagination. I'm just sharing a basic example to help people to get started somewhere.
how does this code jump into next page after extract the current page? thanks
To jump to the next page you will need to add the logic. For the example, the agent will only go to the URL provided and return the dataset for the page.
Are you able to handle complex sites such as Expedia (you have to go past pop-up and “load more” elements) using swarm? Also how about bot detection? Nature websites have anti bot measures in place
This is just a simple use case example. For something more complex like what you described, you will need to incorporate Selenium or Playwright.
In cases where LLM rate limits are low, will this significantly affect the procedure?
I am accessing 4o-mini from a third party service, and their rate limit is 5 requests per minute.
The rate limit won't have much of an impact for the workflow, it is really comes down to input tokes and output token limits.
It's a great resource to get started! Thanks for making that kind of stuff. However, can you please make another video by using Ollama, as it is free to use. I hope you will try..❤
That's actually on one of my to do list using PydanticAI with Ollama models!
@jiejenn waiting for it ❤️✅