Skip to content

Automatically fetch data-xxx attributes #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 12, 2024
Merged

Automatically fetch data-xxx attributes #74

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ea1ff32
Automatically fetch data-xxx attributes
benoit74 Jul 2, 2024
85dc32a
trying out xpath for data- attr findin
ikreymer Jul 12, 2024
9b9ed78
remove duplicate
ikreymer Jul 12, 2024
d34818f
formatting tweaks
ikreymer Jul 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ see [Browsertrix Crawler](https://github.com/webrecorder/browsertrix-crawler) fo

Once the behavior script has been injected, run: `__bx_behaviors.init(opts)` to initialize which behaviors should be used. `opts` includes several boolean options:

- `autofetch` - enable background autofetching of img srcsets, and stylesheets (when possible)
- `autofetch` - enable background autofetching of img srcsets, stylesheets (when possible) and any data-xxx attribute
- `autoplay` - attempt to automatically play and video/audio, or fetch the URLs for any video streams found on the page.
- `autoscroll` - attempt to repeatedly scroll the page to the bottom as far as possible.
- `timeout` - set a timeout (in ms) for all behaviors to finish.
Expand Down
23 changes: 20 additions & 3 deletions src/autofetcher.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
// AutoFetcher script
// extract and fetch all urls from srcsets, from images as well as audio/video
// also extract any urls from media query stylesheets that have not necessarily been loaded
// (May not work for cross-origin stylesheets)
// extract and fetch all urls from
// - srcsets, from images as well as audio/video
// - media query stylesheets that have not necessarily been loaded (may not work for cross-origin stylesheets)
// - any data-xxx attribute

import { BackgroundBehavior } from "./lib/behavior";
import { sleep } from "./lib/utils";
Expand Down Expand Up @@ -83,6 +84,7 @@ export class AutoFetcher extends BackgroundBehavior {

this.extractSrcSrcSetAll(document);
this.extractStyleSheets();
this.extractDataAttributes(document);
}

isValidUrl(url: string) {
Expand Down Expand Up @@ -327,5 +329,20 @@ export class AutoFetcher extends BackgroundBehavior {

text.replace(STYLE_REGEX, urlExtractor).replace(IMPORT_REGEX, urlExtractor);
}

extractDataAttributes(document) {
const allElements = document.querySelectorAll('*');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CSS selectors don't allow querying by attribute name start, but Xpath does!

I think it might be more efficient to do this with xpath, for example, the following snippet can be pasted into the browser.

function* xpathNodes(path, root) {
  root = root || document;
  let iter = document.evaluate(path, root, null, XPathResult.ORDERED_NODE_ITERATOR_TYPE);
  let result = null;
  while ((result = iter.iterateNext()) !== null) {
    yield result;
  }
}

for (const res of xpathNodes("//@*[starts-with(name(), 'data-') and (starts-with(., 'http') or starts-with(., '/') or starts-with(., './') or starts-with(., '../'))]")) {
    console.log(res.value);
}

The xpathNodes is already defined and can be imported from utils

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ended up implementing this to try it out, pushed to the PR!


for (const element of allElements) {
for (const attribute of element.attributes) {
if (attribute.name.startsWith('data-') &&
(attribute.value.startsWith('http') || attribute.value.startsWith('/') || attribute.value.startsWith('./') || attribute.value.startsWith('../'))) {
this.queueUrl(attribute.value);
}
}
}

}

}