Search results

WARC fields not populated in Kafka crawl log

bug

Status: Open.

#90 In ukwa/ukwa-heritrix;

· anjackson opened on Aug 29, 2023

Crawler ignoring robots.txt on a particular site

bug

Status: Open.

#89 In ukwa/ukwa-heritrix;

· anjackson opened on Aug 29, 2023

Add unit test for difficult sitemap case

Status: Open.

#88 In ukwa/ukwa-heritrix;

· anjackson opened on Jun 8, 2023

Timsort comparison error for specific robots.txt URL

Status: Open.

#86 In ukwa/ukwa-heritrix;

· anjackson opened on Nov 28, 2022

Domain crawl tripping 'abuse' alerts

enhancement

Status: Open.

#84 In ukwa/ukwa-heritrix;

· anjackson opened on Oct 10, 2022

Firewall issues when crawling some websites

enhancement

Status: Open.

#83 In ukwa/ukwa-heritrix;

· CarlosR122 opened on Aug 2, 2022

Create a url-frontier Frontier implementation

Status: Open.

#80 In ukwa/ukwa-heritrix;

· anjackson opened on Feb 16, 2022

Add faster/parallel queues for known CDNs

Status: Open.

#79 In ukwa/ukwa-heritrix;

· anjackson opened on Jan 5, 2022

Add support for Redis-compatible alternatives like KvRocks

Status: Open.

#78 In ukwa/ukwa-heritrix;

· anjackson opened on Sep 30, 2021

Verify Redis implementation

Status: Open.

#77 In ukwa/ukwa-heritrix;

· anjackson opened on Sep 30, 2021

Quotas reset on restart

Status: Open.

#76 In ukwa/ukwa-heritrix;

· anjackson opened on Sep 29, 2021

Add support for configuring the Redis DB via an environment variable

Status: Open.

#75 In ukwa/ukwa-heritrix;

· anjackson opened on Sep 29, 2021