Description
URL lists and sitemaps can be crawled in bulk from the command line. It is useful for developers and site operators checking links, collecting pages, or feeding later analysis workflows.
Crawlers can create unwanted traffic or collect content you should not store. Respect site rules, rate limits, credentials, and privacy before scanning large URL sets.