-
Notifications
You must be signed in to change notification settings - Fork 18
Open
Description
Add flag to ignore scraping same URL but with different params.
For example, while scraping a website https://abc.com
, the flag will disable scraping both https://abc.com/something.php?lang=en
and https://abc.com/something.php?lang=ru
, since they are the same page but with different params.
Thanks!
s0rg and kauanmn
Metadata
Metadata
Assignees
Labels
No labels