Page Load Failed due to net::ERR_NAME_NOT_RESOLVED

[prototyperspective]

When running docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl --url [URL] --generateWACZ --text --collection test it doesn't load and the resulting archived page from the wacz file will just show "Archived Page Not Found Sorry, this page was not found in this archive:".

In the output there is this: "message":"Request failed","details":{"url":"…","errorText":"net::ERR_NAME_NOT_RESOLVED"

It seems to be a DNS problem. How to solve this? Is this maybe a known issue when using a VPN? I think DNS over HTTPS is already used since I set cloudflare DNS resolver and DNS works fine.

[atomotic]

I have encountered the same problem, but I have not completely able to debug it.
The bizzarre thing is that if I run and enter the container:

docker run --rm -it -it webrecorder/browsertrix-crawler /bin/bash

and the run the crawl, it works

/usr/bin/crawl --url URL ....

[ikreymer]

This doesn't seem like a Browsertrix specific issue, but perhaps running Docker containers with access to the VPN. You can try --networking=host mode to see if that works better.

[mort666]

This seems to be a problem with brave.. and other chromium browsers on Linux. I get it a huge about from both chromium and brave on the desktop which I'm running Linux. It is common annoyance, has been getting progressively worse over the last few months.

I also get it a lot with the crawler too. I think personally is Linux DNS problem as a result of systemd-resolve and systemd-resolvconf and chrome messing up the resolv.conf as I've noticed sometimes the resolv.conf gets messed up with missing entries.

The easiest workaround I've found it just running resolvectl flush-caches in the terminal as it seems to 'fix' everything till next time which seems to tally with the TTL on the DHCP leases and mDNS ttls for machines affected.

Should also say, the 'archivebox' setup that uses Pihole in the 'stack' to handle DNS, never has this problem.