How does Spider behave with massive Websites (e.g. Fiverr)

[techwizzdom]

Hey all,

I was wondering what would happen if I tried to scrape a website like Fiverr, where there are probably hundreds of thousands of different pages. Since it defaults to scrap everything, what would happen?

Additionally, what would happen if I pass the homepage and limit it to 25? Would it just fetch the homepage + random gigs, or would it go first through static pages (about, careers, etc.)?

[j-mendez]

Hey all,

I was wondering what would happen if I tried to scrape a website like Fiverr, where there are probably hundreds of thousands of different pages. Since it defaults to scrap everything, what would happen?

Additionally, what would happen if I pass the homepage and limit it to 25? Would it just fetch the homepage + random gigs, or would it go first through static pages (about, careers, etc.)?

Hi, It would crawl the entire website if it connects successfully on the first request if the website is static - SSR websites I would use a proxy. For the ordering you can test this with any website to see.