Skip to content

Commit b367b44

Browse files
authored
Experiments framework + various updates (#16, #17)
1 parent ea80969 commit b367b44

31 files changed

+689
-526
lines changed

README.md

Lines changed: 79 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,13 @@ To learn more about our pixels, visit: https://duckduckgo.com/duckduckgo-help-pa
99

1010
Note: The effort to define our pixels is on-going. Not all our product repositories will contain pixel definitions.
1111

12+
## Quick Links
13+
- [Setup](#setup)
14+
- [Documenting a pixel](#documenting-a-pixel)
15+
- [Experiment pixels](#experiment-pixels)
16+
- [All other pixels](#all-other-pixels)
17+
- [Validation](#validation)
18+
1219
## Setup
1320
A repository that supports pixel definitions will have a folder setup with roughly the following structure:
1421
```
@@ -20,56 +27,52 @@ RepoSpecificPixelFolder
2027
--> ...
2128
--> other_pixels.json [file for any pixels that do not belong to a feature folder]
2229
--> ...
23-
--> common_params.json [file that defines commonly used parameters]
24-
--> common_suffixes.json [file that defines commonly used suffixes]
30+
--> params_dictionary.json [file that defines commonly used parameters]
31+
--> suffixes_dictionary.json [file that defines commonly used suffixes]
32+
--> native_experiments.json [file that defines pixels sent by the native experiments framework]
2533
```
2634

2735
You can organize the files and sub-directories within `pixels` however you like, the example above is just one option.
2836

29-
## Validation
30-
**Background**:
31-
* Validation ensures that pixel definitions conform to the [schema](./schemas/pixel_schema.json5) and follow a consistent format.
32-
* Validation will run as part of CI, but you can also run it manually - details below.
33-
* A repository that supports pixel definitions will have a folder setup with `package.json` pointing to this module, referred to as `PackageFolder` below.
34-
* Note: usually `PackageFolder` is the same as the `RepoSpecificPixelFolder` referenced in the previous section.
35-
36-
**Pre-requisites**:
37-
* Install Node.js: see instructions in https://nodejs.org/en/download
38-
* Install dependencies:
39-
```
40-
$ cd ${PackageFolder}
41-
$ npm i
42-
```
43-
44-
**Running validation**:
45-
```
46-
$ cd ${PackageFolder}
47-
$ npm run validate-defs
48-
```
49-
Note:
50-
* If formatting errors are found, you can fix them with `npm run lint.fix`
51-
* For schema validation failures, check the output and apply fixes manually
52-
* You can also (re)validate a single file:
53-
* Schema validation: `npx validate-ddg-pixel-defs . -f ${path to file relative to PackageFolder/pixels/ directory}`
54-
* Formatting: `npx prettier ${path to file relative to PackageFolder/ directory} --check`
55-
5637
## Documenting a pixel
57-
Each JSON file can contain multiple pixels, keyed by the static portion of the pixel name.
58-
Add your pixel where it makes the most sense.
59-
60-
You can use either JSON or JSON5 (JSON with comments, trailing commas) to document your pixels.
61-
All pixel definitions must adhere to the [pixel schemas](./schemas/pixel_schema.json5).
38+
### Experiment pixels
39+
Pixels sent by the [native experiments framework](https://app.asana.com/1/137249556945/project/1208889145294658/task/1209331148407154?focus=true)
40+
must be documented separately in `native_experiments.json` and adhere to the [native experiments schema](./schemas/native_experiments.schema.json5).
41+
42+
**Fields**:
43+
* `defaultSuffixes`: defines what suffixes are appended to each pixel after the cohort
44+
* Example: Android appends `.android.[phone|tablet]` to its pixels
45+
* See [Pixels with dynamic names](#pixels-with-dynamic-names) for more info on how to define these
46+
* `activeExperiments`: defines each experiment with the key being the name of the experiment. Each experiment must also define:
47+
* `cohorts`: an array of Strings defining each cohort in the experiment
48+
* `metrics`: a collection of objects where each object is keyed by the metric name as it would appear in the pixel. Each metric must also provide:
49+
* `description` of the metric
50+
* `enum` of possible values
51+
52+
**Note**: The following are pre-defined and are automatically taken into account by the pixel schema
53+
(you do not need to worry about defining them):
54+
* `enrollmentDate` and `conversionWindowDays` parameters
55+
* `app_use` and `search` metrics
56+
57+
**Example definition**: [native_experiments.json](./tests/test_data/valid/native_experiments.json)
58+
59+
### All other pixels
60+
* All other pixels must be defined in any file within the `pixels` directory and its children
61+
* Each JSON file can contain multiple pixels, keyed by the static portion of the pixel name
62+
* Add your pixel where it makes the most sense
63+
* You can use either JSON or JSON5 (JSON with comments, trailing commas) to document your pixels
64+
* Pixel definitions must adhere to the [pixel schema](./schemas/pixel_schema.json5)
6265

6366
Below, you'll find a walkthrough of the schema requirements and options.
6467
As you read through, you can refer to the [pixel_guide.json](./tests/test_data/valid/pixels/pixel_guide.json5) for examples.
6568

66-
### Minimum requirements
69+
#### Minimum requirements
6770
Each pixel **must** contain the following properties:
6871
* `description` - when the pixel fires and its purpose
6972
* `owners` - DDG usernames of who to contact about the pixel
7073
* `triggers` - one or more of the [possible triggers](./schemas/pixel_schema.json5#27) that apply to the pixel
7174

72-
### Pixels with dynamic names
75+
#### Pixels with dynamic names
7376
If the pixel name is parameterized, you can utilize the `suffixes` property.
7477

7578
Required properties for each suffix:
@@ -79,7 +82,13 @@ Optional properties for each suffix:
7982
* `key` - static portion of the suffix
8083
* JSON schema types - used to indicate constrained values for the suffix. Can be anything from https://json-schema.org/understanding-json-schema/reference/type
8184

82-
### Pixels with parameters
85+
Note:
86+
* You can utilize a 'shortcut' to point to a common suffix that's predefined in `suffixes_dictionary.json`
87+
* See `device_type` in [pixel_guide.json](./tests/test_data/valid/pixels/pixel_guide.json5)
88+
and [suffixes_dictionary.json](./tests/test_data/valid/suffixes_dictionary.json)
89+
* Ordering of suffixes matters
90+
91+
#### Pixels with parameters
8392
If the pixel contains parameters, you can utilize the `parameters` property.
8493

8594
Required properties for each parameter:
@@ -91,9 +100,41 @@ Required properties for each parameter:
91100
Optional properties for each parameter:
92101
* JSON schema types - used to indicate constrained parameter values. Can be anything from https://json-schema.org/understanding-json-schema/reference/type
93102

94-
### Temporary pixels
103+
* You can utilize a 'shortcut' to point to a common parameter that's predefined in `params_dictionary.json`
104+
* See `appVersion` in [pixel_guide.json](./tests/test_data/valid/pixels/pixel_guide.json5)
105+
and [params_dictionary.json](./tests/test_data/valid/params_dictionary.json)
106+
* Ordering of suffixes matters
107+
108+
#### Temporary pixels
95109
If the pixel is temporary, set an expiration date in the `expires` property.
96110

111+
## Validation
112+
**Background**:
113+
* Validation ensures that pixel definitions conform to the [schema](./schemas/pixel_schema.json5) and follow a consistent format.
114+
* Validation will run as part of CI, but you can also run it manually - details below.
115+
* A repository that supports pixel definitions will have a folder setup with `package.json` pointing to this module, referred to as `PackageFolder` below.
116+
* Note: usually `PackageFolder` is the same as the `RepoSpecificPixelFolder` referenced in the previous section.
117+
118+
**Pre-requisites**:
119+
* Install Node.js: see instructions in https://nodejs.org/en/download
120+
* Install dependencies:
121+
```
122+
$ cd ${PackageFolder}
123+
$ npm i
124+
```
125+
126+
**Running validation**:
127+
```
128+
$ cd ${PackageFolder}
129+
$ npm run validate-defs
130+
```
131+
Note:
132+
* If formatting errors are found, you can fix them with `npm run lint.fix`
133+
* For schema validation failures, check the output and apply fixes manually
134+
* You can also (re)validate a single file:
135+
* Schema validation: `npx validate-ddg-pixel-defs . -f ${path to file relative to PackageFolder/pixels/ directory}`
136+
* Formatting: `npx prettier ${path to file relative to PackageFolder/ directory} --check`
137+
97138
## License
98139
DuckDuckGo Pixels Schema is distributed under the [Apache 2.0 License](LICENSE).
99140

bin/validate_schema.mjs

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,11 +37,20 @@ const mainDir = argv.dirPath;
3737
const pixelsDir = path.join(mainDir, 'pixels');
3838
const commonParams = fileUtils.readCommonParams(mainDir);
3939
const commonSuffixes = fileUtils.readCommonSuffixes(mainDir);
40-
const validator = new DefinitionsValidator(commonParams, commonSuffixes);
40+
const pixelIgnoreParams = fileUtils.readIgnoreParams(mainDir);
41+
const globalIgnoreParams = fileUtils.readIgnoreParams(fileUtils.GLOBAL_PIXEL_DIR);
42+
const ignoreParams = { ...pixelIgnoreParams, ...globalIgnoreParams };
43+
44+
const validator = new DefinitionsValidator(commonParams, commonSuffixes, ignoreParams);
4145
logErrors('ERROR in common_params.json:', validator.validateCommonParamsDefinition());
4246
logErrors('ERROR in common_suffixes.json:', validator.validateCommonSuffixesDefinition());
47+
logErrors('ERROR in ignore_params.json:', validator.validateIgnoreParamsDefinition());
48+
49+
// 2) Validate experiments
50+
const experiments = fileUtils.readExperimentsDef(mainDir);
51+
logErrors('ERROR in native_experiments.json:', validator.validateExperimentsDefinition(experiments));
4352

44-
// 2) Validate pixels and params
53+
// 3) Validate pixels and params
4554
function validateFile(file) {
4655
console.log(`Validating pixels definition: ${file}`);
4756
const pixelsDef = JSON5.parse(fs.readFileSync(file));

global_pixel_definitions/ignore_params.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"kp": {
33
"key": "kp",
44
"description": "",
5-
"enum": [1]
5+
"enum": [1, -1, 0]
66
},
77
"p": {
88
"key": "p",
@@ -17,6 +17,6 @@
1717
"scrlybrkr": {
1818
"key": "scrlybrkr",
1919
"description": "Added by 3rd party school proxy",
20-
"type": "number"
20+
"pattern": "^[0-9a-fA-F]+$"
2121
}
2222
}

live_validation_scripts/validate_live_pixel.mjs

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,30 +2,32 @@
22

33
import csv from 'csv-parser';
44
import fs from 'fs';
5+
import JSON5 from 'json5';
56

67
import { getArgParserWithCsv } from '../src/args_utils.mjs';
78
import { ParamsValidator } from '../src/params_validator.mjs';
89
import { LivePixelsValidator } from '../src/live_pixel_validator.mjs';
910

1011
import * as fileUtils from '../src/file_utils.mjs';
12+
import { PIXEL_DELIMITER } from '../src/constants.mjs';
1113

1214
const argv = getArgParserWithCsv('Validates pixels from the provided CSV file', 'path to CSV file containing pixels to validate').parse();
1315

1416
function main(mainDir, csvFile) {
1517
console.log(`Validating live pixels in ${csvFile} against definitions from ${mainDir}`);
1618

1719
const productDef = fileUtils.readProductDef(mainDir);
20+
const experimentsDef = fileUtils.readExperimentsDef(mainDir);
1821
const commonParams = fileUtils.readCommonParams(mainDir);
1922
const commonSuffixes = fileUtils.readCommonSuffixes(mainDir);
20-
const globalIgnoreParams = fileUtils.readIgnoreParams(fileUtils.GLOBAL_PIXEL_DIR);
21-
2223
const tokenizedPixels = fileUtils.readTokenizedPixels(mainDir);
23-
const paramsValidator = new ParamsValidator(commonParams, commonSuffixes);
24-
const pixelIgnoreParams = fileUtils.readIgnoreParams(mainDir);
2524

25+
const pixelIgnoreParams = fileUtils.readIgnoreParams(mainDir);
26+
const globalIgnoreParams = fileUtils.readIgnoreParams(fileUtils.GLOBAL_PIXEL_DIR);
2627
const ignoreParams = [...(Object.values(pixelIgnoreParams) || []), ...Object.values(globalIgnoreParams)];
28+
const paramsValidator = new ParamsValidator(commonParams, commonSuffixes, ignoreParams);
2729

28-
const liveValidator = new LivePixelsValidator(tokenizedPixels, productDef, ignoreParams, paramsValidator);
30+
const liveValidator = new LivePixelsValidator(tokenizedPixels, productDef, experimentsDef, paramsValidator);
2931
let processedPixels = 0;
3032
fs.createReadStream(csvFile)
3133
.pipe(csv())
@@ -34,7 +36,9 @@ function main(mainDir, csvFile) {
3436
if (processedPixels % 100000 === 0) {
3537
console.log(`...Processing row ${processedPixels.toLocaleString('en-US')}...`);
3638
}
37-
liveValidator.validatePixel(row.pixel, row.params);
39+
const pixelRequestFormat = row.pixel.replaceAll('.', PIXEL_DELIMITER);
40+
const paramsUrlFormat = JSON5.parse(row.params).join('&');
41+
liveValidator.validatePixel(pixelRequestFormat, paramsUrlFormat);
3842
})
3943
.on('end', async () => {
4044
console.log(`\nDone.\nTotal pixels processed: ${processedPixels.toLocaleString('en-US')}`);

main.mjs

Lines changed: 9 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -11,18 +11,18 @@ import { tokenizePixelDefs } from './src/tokenizer.mjs';
1111
*/
1212

1313
/**
14-
* Given a pixels dir, build a LivePixelsValidator. Optionally, override values on disk with
15-
* provided overrides.
14+
* Build a LivePixelsValidator
1615
* @param {object} commonParams
1716
* @param {object} commonSuffixes
1817
* @param {ProductDefinition} productDef
1918
* @param {object} ignoreParams
2019
* @param {object} tokenizedPixels
20+
* @param {object} experimentsDef
2121
* @returns
2222
*/
23-
export function buildLivePixelValidator(commonParams, commonSuffixes, productDef, ignoreParams, tokenizedPixels) {
24-
const paramsValidator = new ParamsValidator(commonParams, commonSuffixes);
25-
return new LivePixelsValidator(tokenizedPixels, productDef, ignoreParams, paramsValidator);
23+
export function buildLivePixelValidator(commonParams, commonSuffixes, productDef, ignoreParams, tokenizedPixels, experimentsDef = {}) {
24+
const paramsValidator = new ParamsValidator(commonParams, commonSuffixes, ignoreParams);
25+
return new LivePixelsValidator(tokenizedPixels, productDef, experimentsDef, paramsValidator);
2626
}
2727

2828
/**
@@ -46,21 +46,16 @@ export function buildTokenizedPixels(allPixelDefs) {
4646
export function validateSinglePixel(validator, url) {
4747
const parsedUrl = new URL(url);
4848
// parse pixel ID out of the URL path
49-
const pixel = parsedUrl.pathname.slice(3).replaceAll('_', '.');
50-
// validator expects a JSON encoded array of parameters
51-
const params = JSON.stringify(
52-
parsedUrl.search
53-
.slice(1)
54-
.split('&')
55-
.filter((v) => !v.match(/^\d+$/)),
56-
);
49+
const pixel = parsedUrl.pathname.slice(3);
50+
// validator expects URL params after cache buster
51+
const params = parsedUrl.search.slice(1).replace(/^\d+=?&/, '');
5752
// reset errors in validator
5853
validator.pixelErrors = {};
5954
validator.undocumentedPixels.clear();
6055
// validate
6156
validator.validatePixel(pixel, params);
6257
if (validator.undocumentedPixels.size > 0) {
63-
throw new Error(`Undocumented Pixel: ${JSON.stringify(validator.undocumentedPixels)}`);
58+
throw new Error(`Undocumented Pixel: ${JSON.stringify(Array.from(validator.undocumentedPixels))}`);
6459
}
6560
if (Object.keys(validator.pixelErrors).length > 0) {
6661
throw new Error(`Pixel Errors: ${JSON.stringify(validator.pixelErrors)}`);

0 commit comments

Comments
 (0)