Don't do direct fetch for every page URL

[ikreymer]

Can add some simple optimizations to only try direct fetch of pages with certain known extensions, otherwise, assume HTML.
This avoids the extra fetch that happens for every page (and may contribute to rate limits).

[fservida]

I would advise against this. It is quite common for files to be generated on the fly or hidden behind urls with no extension, mime type checking is generally the best option.

If avoiding that initial fetch is really important I would suggest keeping the current behaviour as an option.