Test crawler performance

[SebastianZimmeck]

Before we start the crawl, we need to test the crawler's performance. So, we need to compare the manually observed groundtruth with the analysis results. We probably need a 100-site test set.

• How do we select the test set sites given the different locations and states (issue Create Manually Curated List of Sites to Crawl #7) so that we have good test coverage?
• One issue is that different loads of a site may lead to different trackers etc. detected. So, we need to look for the groundtruth and analysis results at exactly the same site load. So, maybe, just load one site, get both groundtruth and analysis results and check?
• We need to document all of that

(@JoeChampeau and @jjeancharles feel free to participate here as well.)

[SebastianZimmeck]

Where are we with the testing protocol, @danielgoldelman?

[danielgoldelman]

Preliminary testing protocol

1. Run the crawl, collect the data.

2. Separate the pp data from the entries data.

For pp data:

1. Create spreadsheet for each root url

2. Log every piece of data into the spreadsheet with everything pp gives us, separated by pp data type

For all http request data:


1. Create spreadsheet for each root url

2. Do the most generic string matching with the values we are looking for. Note: we will have lists of keywords per vpn, we can get the ipinfo location while using the vpn by going to their site, and we can find monetization labels within the http requests. EX: if the zip code should be 10001, instead of a regex of \D10001\D, we look for just the string 10001. For every single key we could be looking for, we run it on the http requests gathered. Coallate these possible data stealing requests

3. Go through every http request and label, adding to the spreadsheet when necessary

Now to bring both together:


1. We have the two spreadsheet documents now. Time to classify

2. Potentially in a new spreadsheet, place all http requests that occur in both pp and all http requests first. Then all that only occurred in pp. then all that only occurred in the http requests.
3. Perform classification

[SebastianZimmeck]

@danielgoldelman, can you reformat and reorder your comment? The order is very hard to follow, there are multiple numbers 1 and 2 after each other, etc.

[danielgoldelman]

@SebastianZimmeck sorry, the original comment was written on GitHub mobile, so formatting was hard to check. Changes made above.

[SebastianZimmeck]

@danielgoldelman and @dadak-dom, please carefully read section 3.2.2 and 3.3 (in particular, 3.3.2) of our paper. We can re-use much of the approach there. I do not think that we need an annotation of the ground truth, but both of you should check the ground truth (for whatever our definition of ground truth is) and come to the same conclusion.

We have to create a testing protocol along the lines of the following:

1. Select the set of analysis functionality that we are testing and how

   • By default all analysis functionalities
   • But how are we going to test for keywords, for example? How for email addresses, how for phone numbers, ...?

2. Pick a set of websites to test

   • How many? Probably, 100 to 200. We need some reasonable standard deviation. For example, it is meaningless to test a particular analysis functionality for just one site because a successful test would not allow us to extrapolate and claim that we are successful for, say, 1,000 sites in our crawl set with that functionality. So, we need, say, 10 sites successfully analyzed to make that claim. Can you solidify that? What is the statistical significance of 10 sites? @JoeChampeau can help with the statistics. We should have some statistical power along the lines of "with 95% confidence our analysis of latitude is within the bounds of a 10% error rate" (e.g., if we detect 1,000 sites having a latitude with 95% confidence the real result is between 900 and 1,100 sites).
   • Which sites to select? Again, the selected set should allow us to make the claim that if an analysis functionality works properly on the test set, it also works for the large set of sites that we crawl. So, we would need to pick a diverse set of sites covering every analysis functionality for each region that we cover. There should be no bias. For example, there will be no problems for monetization categories because they occur so frequently, but how do we ensure, e.g., that there is a meaningful number of sites that collect latitudes? Maybe, pick map sites from somewhere? How do we pick sites for keywords (assuming we are analyzing keywords)?
   • Are we using the same test set of sites for each country/state? Yes, no, is some overlap OK, is it harmful, is it good, ...?
   • How are we selecting sites randomly? Use random.org.
   • We can't select any sites that we used for preliminary testing, i.e., validation. So, which are the sites, if any, that need to be excluded? If we randomly select an excluded site, how do we pick a new one? Maybe, just the next one on a given list.

3. Running the test

   • Are we testing one site a time or run the complete test set? If we do the former we need to record all site data (and be absolutely sure that there are no errors and nothing omitted in the recording). We need to get both the analysis results and the ground truth data at the same time. The reason is that when we load a site multiple times, there is a good chance that not all trackers and other network connections are identical for both loads. So, the analysis results could diverge from the ground truth if the latter is based on a different load. We need to check the ground truth for the exact site load from which we got the analysis results. The alternative to a complete test set crawl is to do the analysis for one site at a time, i.e., visit a site, record the PP analysis results, use browser developer tools (and other tools, as necessary) to check the ground truth, record the evidence, record the ground truth evidence and result, then analyze the next site and so on. So, we would be doing multiple crawls of one site.
   • We will also need to change the VPN for every different location.
   • Who is going to run the test? @JoeChampeau has the computer. Is it you, @dadak-dom or @danielgoldelman? Both the PP analysis results and the ground truth should be checked by two people independently. This seems easier if only one test set crawl is done as opposed to the site-by-site approach.

4. Ground truth analysis

   • How do we analyze the ground truth? Per your comment above, @danielgoldelman, I'll take that we do string matching in HTTP messages. Is that a reliable indicator? Maybe, we would also need to look at, say, browser API usage for latitude, i.e., the browser prompting the user to allow location access. What are the criteria to reliably analyze the ground truth? This can be different for our different functionalities.

These question cannot be answered in the abstract. @danielgoldelman and @dadak-dom, please play around with some sites for each analysis functionality and come up with a protocol to analyze it. For each functionality you need to be convinced that you can reliably identify true positives (and exclude false positives and false negatives). In other words, please do some validation tests.

[dadak-dom]

Who is going to run the test?

Would it make sense if @JoeChampeau runs the test, and then hands the data over to Daniel and me? I thought it would make sense since that's the computer that we will use to run the actual crawl. That way, we could avoid any potential issues arising when switching between windows and mac. Just a thought.

@SebastianZimmeck , the way I understand it, we will end up with three different site lists for each country (please correct me if I'm wrong)

1. Validation (what Daniel and I are doing now)
2. Test set (what we're preparing for and will soon be running)
3. The actual crawl list.
   We cannot have any overlap between the validation and the test set, but can the test set (and/or the validation) be derived from the actual crawl list? I would need to know this before I start making any lists for the test set.

[SebastianZimmeck]

Would it make sense if @JoeChampeau runs the test, and then hands the data over to Daniel and me?

It certainly makes sense, but that would depend on if @JoeChampeau has time as the task was originally @danielgoldelman's. (Given our slow speed, the point may more or less resolve itself since we will be all back on campus soon anyways.)

(please correct me if I'm wrong)

All correct.

but can the test set (and/or the validation) be derived from the actual crawl list?

Yes, the validation and test set can be derived from the crawl list.

[dadak-dom]

I have added my proposed crawl testing lists to the branch connected with this issue (issue-9). Here was my procedure:

1. For each country that we will crawl, create a new .csv file.
2. Go to random.org and have it generate a list of random integers from 1-525
3. Take the first six integers and find the matching URL from the general list
4. Regenerate the random integers and find the six matching URLs from the country specific list
5. If there seems to be a bias for one functionality, throw the list out and try again. (Or, if there is any overlap with sites that were used for validation. Luckily, this was never the case for me)
6. Repeat the process for each location we will crawl, so ten times total.

With point 5 I tried my best to include a fair share of sites that take locations, as monetization was easy to come by. @SebastianZimmeck let me know if any changes need to be made.

[SebastianZimmeck]

OK, sounds good!

So, our test set has a total of 120 sites? For each of the 10 countries/states 6 sites from the general list and 6 from the country-specific list.

With point 5 I tried my best to include a fair share of sites that take locations

How did you make the guess that a site takes locations?

[dadak-dom]

So, our test set has a total of 120 sites? For each of the 10 countries/states 6 sites from the general list and 6 from the country-specific list.

Yes, 120 sites total.

How did you make the guess that a site takes locations?

A couple of ways, e.g. visiting the site and seeing if it requests the location from the browser, or if PP detects a location, or if I know from my own browsing that the site would take locations.