Importing browser user_data_dir generated with Selenium for crawl

[msramalho]

I've been experimenting with Selenium and SeleniumBase to access auth-walled content with good results, one of the strategies is providing the user-data-dir/user_data_dir to indicate a directory for the default chrome profile to be stored in, this allows reusing the sessions for quite some time. Some of the websites need more than just the cookies to accept the login (I'm thinking local storage and maybe more).

Mainly: As it stands, is there any way of importing this same profile into browsertrix-crawler? Is the --profile option in browsertrix compatible with this directory or interchangeable somehow?

extra:

• I would think Chrome (from Selenium) -> Brave profiles may be compatible but have not tested it.
• I'd also be curious to see if this could be done on the create-login-profile stage, and even if it is possible to connect Selenium/Selenium base to the VNC server.

[ikreymer]

Yes, I believe that profiles created with same or older version of Chrome/Chromium that is used in the current Brave version should work. Each Brave release lists the current version of Chromium so you should make sure your profile is from a version <= then that. The --profile should point to a tar.gz of the profile directory, so I think it should work.

More generally, curious what is the functionality that you're using in Selenium? Can it be automated in puppeteer as well, and done directly here, for example, with a custom script? The profiles include localStorage and we just added a feature to support crawling localStorage #856 as an option.