Skip to content

Commit 8e1c2c6

Browse files
library-testinglibrary-testing
committed
first commit
0 parents  commit 8e1c2c6

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+5411
-0
lines changed

‎.gitignore

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# Created by https://www.toptal.com/developers/gitignore/api/python,visualstudiocode
2+
# Edit at https://www.toptal.com/developers/gitignore?templates=python,visualstudiocode
3+
4+
### Model ###
5+
.vscode/
6+
.git/
7+
data/*_output
8+
*.db
9+
temp/
10+
*.npz
11+
*.npy
12+
*.csv
13+
14+
15+
### Python ###
16+
# Byte-compiled / optimized / DLL files
17+
__pycache__/
18+
*.py[cod]
19+
*$py.class
20+
21+
# C extensions
22+
*.so
23+
24+
# Distribution / packaging
25+
.Python
26+
build/
27+
develop-eggs/
28+
dist/
29+
downloads/
30+
eggs/
31+
.eggs/
32+
parts/
33+
sdist/
34+
var/
35+
wheels/
36+
pip-wheel-metadata/
37+
share/python-wheels/
38+
*.egg-info/
39+
.installed.cfg
40+
*.egg
41+
MANIFEST
42+
43+
# PyInstaller
44+
# Usually these files are written by a python script from a template
45+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
46+
*.manifest
47+
*.spec
48+
49+
# Installer logs
50+
pip-log.txt
51+
pip-delete-this-directory.txt
52+
53+
# Unit test / coverage reports
54+
htmlcov/
55+
.tox/
56+
.nox/
57+
.coverage
58+
.coverage.*
59+
.cache
60+
nosetests.xml
61+
coverage.xml
62+
*.cover
63+
*.py,cover
64+
.hypothesis/
65+
.pytest_cache/
66+
pytestdebug.log
67+
68+
# Translations
69+
*.mo
70+
*.pot
71+
72+
# Django stuff:
73+
*.log
74+
local_settings.py
75+
db.sqlite3
76+
db.sqlite3-journal
77+
78+
# Flask stuff:
79+
instance/
80+
.webassets-cache
81+
82+
# Scrapy stuff:
83+
.scrapy
84+
85+
# Sphinx documentation
86+
docs/_build/
87+
doc/_build/
88+
89+
# PyBuilder
90+
target/
91+
92+
# Jupyter Notebook
93+
.ipynb_checkpoints
94+
95+
# IPython
96+
profile_default/
97+
ipython_config.py
98+
99+
# pyenv
100+
.python-version
101+
102+
# pipenv
103+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
104+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
105+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
106+
# install all needed dependencies.
107+
#Pipfile.lock
108+
109+
# poetry
110+
#poetry.lock
111+
112+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
113+
__pypackages__/
114+
115+
# Celery stuff
116+
celerybeat-schedule
117+
celerybeat.pid
118+
119+
# SageMath parsed files
120+
*.sage.py
121+
122+
# Environments
123+
# .env
124+
.env/
125+
.venv/
126+
env/
127+
venv/
128+
ENV/
129+
env.bak/
130+
venv.bak/
131+
pythonenv*
132+
133+
# Spyder project settings
134+
.spyderproject
135+
.spyproject
136+
137+
# Rope project settings
138+
.ropeproject
139+
140+
# mkdocs documentation
141+
/site
142+
143+
# mypy
144+
.mypy_cache/
145+
.dmypy.json
146+
dmypy.json
147+
148+
# Pyre type checker
149+
.pyre/
150+
151+
# pytype static type analyzer
152+
.pytype/
153+
154+
# operating system-related files
155+
# file properties cache/storage on macOS
156+
*.DS_Store
157+
# thumbnail cache on Windows
158+
Thumbs.db
159+
160+
# profiling data
161+
.prof
162+
163+
164+
### VisualStudioCode ###
165+
.vscode/*
166+
!.vscode/settings.json
167+
!.vscode/tasks.json
168+
!.vscode/launch.json
169+
!.vscode/extensions.json
170+
*.code-workspace
171+
172+
### VisualStudioCode Patch ###
173+
# Ignore all local history of files
174+
.history
175+
.ionide
176+
177+
# End of https://www.toptal.com/developers/gitignore/api/python,visualstudiocode

‎README.md

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing
2+
3+
4+
This is the implementation repository of our *ICSE'22* paper: **Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing**.
5+
6+
7+
8+
## Description
9+
10+
Deep learning (DL) techniques are shown to be effective in many challenging tasks, and are hence widely-adopted in practice. However, previous work has shown that DL libraries, the basis of building and executing DL models, contain bugs and can cause severe consequences. Unfortunately, existing approaches still cannot comprehensively excise DL libraries. They utilize existing trained models and only detect bugs in model inference phase. In this work we propose ***Muffin*** to address these issues. To this end, *Muffin* applies a specifically-designed model fuzzing approach, which allows it to generate diverse DL models to explore the target library, instead of relying only on existing trained models. *Muffin* makes differential testing feasible in the model training phase by tailoring a set of metrics to measure the inconsistency between different DL libraries. In this way, *Muffin* can best excise the library code to detect more bugs. Experiments on three widely-used DL libraries show that *Muffin* can detect **39 new bugs**.
11+
12+
You can access this repository using the following command:
13+
```shell
14+
git clone https://github.com/library-testing/Muffin.git
15+
```
16+
17+
18+
19+
## Libraries
20+
21+
We use three widely-used DL libraries (*i.e.*, ***TensorFlow***, ***Theano***, and ***CNTK***) as the backend low-level libraries as our testing targets, and ***Keras*** as the frontend high-level library. To sufficiently illustrate the effectiveness of ***Muffin***, we utilize a total of **15 release versions** of the three backend libraries, and construct five experimental environments for differential testing as follow:
22+
23+
| ID | Keras | TensorFlow | Theano | CNTK |
24+
| ---- | ----- | ---------- | ------ | ----- |
25+
| E1 | 2.3.1 | 2.0.0 | 1.0.4 | 2.7.0 |
26+
| E2 | 2.3.1 | 1.15.0 | 1.0.3 | 2.6.0 |
27+
| E3 | 2.2.4 | 1.12.0 | 1.0.2 | 2.5.0 |
28+
| E4 | 2.2.4 | 1.11.0 | 1.0.1 | 2.4.0 |
29+
| E5 | 2.2.4 | 1.10.0 | 1.0.0 | 2.3.0 |
30+
31+
In order to facilitate other researchers to reproduce ***Muffin***, we provide ***docker*** images for each experiments (*i.e.*, E1 ~ E5), named `librarytesting/muffin` with tags from `E1` to `E5` respectively.
32+
33+
34+
35+
## Datasets(Optional)
36+
37+
Our approach is not sensitive to datasets, *i.e.*, theoretically any data type can be used for testing. So you can just test with **randomly generated dataset** with our source code, for briefness.
38+
39+
If you want to do comparative experiments with existing approaches, 6 widely-used datasets mentioned in our paper can be used, *i.e.*, **MNIST**, **F-MNIST**, **CIFAR-10**, **ImageNet**, **Sine-Wave** and **Stock-Price**. The first three ones can be accessed by [Keras API](https://keras.io/api/datasets/)，while the rest can be access from [OneDrive](https://onedrive.live.com/?authkey=%21ANVR8C2wSN1Rb9M&id=34CB15091B189D3E%211909&cid=34CB15091B189D3E)(`dataset.zip`) provided by [LEMON](https://github.com/Jacob-yen/LEMON).
40+
41+
42+
43+
## Environment
44+
45+
As we mentioned above, ***docker*** images are provided for experiments.
46+
47+
* **Step 0:** Please install ***nvidia-docker2***. You can install it by following this [document](https://codepyre.com/2019/01/installing-nvidia-docker2-on-ubuntu-18.0.4/).
48+
* **Step 1:** Clone this repository into `[PATH_TO_MUFFIN]`. If you want to use the last three dataset mentioned above, download and unzip them into `[PATH_TO_MUFFIN]/dataset`.
49+
50+
`[PATH_TO_MUFFIN]` is the local path you want to `git clone` this repository into.
51+
52+
* **Step 2:** Use the following command to pull the ***docker*** image we released (take `E1` as an example), and create a container for it:
53+
54+
```shell
55+
docker pull librarytesting/muffin:E1
56+
docker run --runtime=nvidia -it -v [PATH_TO_MUFFIN]:/data --name muffin-E1 librarytesting/muffin:E1 /bin/bash
57+
```
58+
59+
At this point, you're inside the ***docker*** container and ready to the experiment.
60+
61+
* **Step 3**: Enter the virtual environment we have set up in the container:
62+
63+
```shell
64+
source activate lemon
65+
```
66+
67+
68+
69+
## Experiments
70+
71+
Make sure you are now in the ***docker*** container!
72+
73+
#### 1. Configuration
74+
75+
A configuration file `testing_config.json` should be provided to flexibly set up testing configuration. Here is an example:
76+
77+
```json
78+
{
79+
"debug_mode": 1,
80+
"dataset_name": "mnist",
81+
"case_num": 200,
82+
"generate_mode": "seq",
83+
"data_dir": "data",
84+
"timeout": 300,
85+
"use_heuristic": 1
86+
}
87+
```
88+
89+
* `debug_mode` can be set between `0` and `1`. `0` represents testing with **randomly generated dataset**, `1` represents testing with existing datasets we mentioned above. In mode `0`, no dataset is required.
90+
* `dataset_name` indicates the name of dataset in mode `1`. The available options include `cifar10`, `mnist`, `fashion_mnist`, `imagenet`, `sinewave`, `price`. If you test in mode `0`, you **should** randomly set a name that does not conflict with the existing name for distinction.
91+
* `case_num` indicates the number of random models to generate for test.
92+
* `generate_mode` indicates the mode of model generation,. The available options include `seq`, `merging`, `dag`, `template`.
93+
* `data_dir` indicates the data directory to store outputs. Remaining `data` is recommended.
94+
* `timeout` indicates the timeout duration for each model.
95+
* `use_heuristic` indicates if using the heuristic method mentioned in the paper or not. `1` is recommended.
96+
97+
#### 2. Preprocessing
98+
99+
* **Dataset:** Execute the following command in `/data/dataset` to preprocess the dataset or downloading them from [Keras API](https://keras.io/api/datasets/) if you want to use existing dataset:
100+
101+
```shell
102+
python get_dataset.py [DATASET_NAME]
103+
```
104+
105+
`[DATASET_NAME]` can be chosen from `cifar10`、 `mnist`、`fashion_mnist`、`imagenet`、`sinewave`、`price`.
106+
107+
* **Generate database:** Execute the following command in `/data/data` to create the ***sqlite*** database for storing the testing results:
108+
109+
```
110+
sqlite3 [DATASET_NAME].db < create_db.sql
111+
```
112+
113+
Remember to change `[DATASET_NAME]` to the name of the dataset you have set in `testing_config.json`.
114+
115+
#### 3. Start
116+
117+
Use the following command to run the experiment according to the configuration:
118+
119+
```shell
120+
python run.py
121+
```
122+
123+
The `testing_config.json` file should be place in the same directory.
124+
125+
The testing results will be store in `/data/[DATA_DIR]/[DATASET_NAME].db` (*e.g.* `/data/data/mnist.db`), and **detail results** for each model will be stored in `/data/[DATA_DIR]/[DATASET_NAME]_output`.
126+
127+
Use the following command in `/data/data` to delete a set of testing results ( **carefully use!** ):
128+
129+
```shell
130+
python clear_data.py [DATASET_NAME]
131+
```
132+
133+
134+
135+
## Citation
136+
137+
Coming soon...
138+

‎data/clear_data.py

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
from pathlib import Path
2+
import shutil
3+
import sqlite3
4+
import sys
5+
6+
dataset_name = sys.argv[1] # e.g. 'fashion_mnist'
7+
8+
output_path = Path(sys.argv[0]).resolve().parent / f'{dataset_name}_output'
9+
if output_path.exists():
10+
shutil.rmtree(output_path)
11+
12+
13+
report_path = Path(sys.argv[0]).resolve().parent / f'{dataset_name}_report'
14+
if report_path.exists():
15+
shutil.rmtree(report_path)
16+
17+
18+
db_path = Path(sys.argv[0]).resolve().parent / f'{dataset_name}.db'
19+
if db_path.exists():
20+
conn = sqlite3.connect(str(db_path), check_same_thread=False)
21+
22+
CLEAR_MODEL = '''drop table model'''
23+
CLEAR_INCONS = '''drop table inconsistency'''
24+
CLEAR_MAP = '''drop table localization_map'''
25+
CREATE_MODEL = '''CREATE TABLE "model" (
26+
"dataset_name" TEXT,
27+
"node_num" INTEGER NOT NULL,
28+
"generate_fail_backends" TEXT,
29+
"crash_backends" TEXT,
30+
"nan_backends" TEXT,
31+
"inf_backends" TEXT,
32+
"loss_func" TEXT,
33+
"optimizer" TEXT,
34+
"status" TEXT
35+
)
36+
'''
37+
CREATE_INCONS = '''CREATE TABLE "inconsistency" (
38+
"model_id" INTEGER NOT NULL,
39+
"backend_pair" TEXT NOT NULL,
40+
"model_output_delta" REAL,
41+
"loss_delta" REAL,
42+
"loss_grads_delta" REAL,
43+
"weights_delta" REAL,
44+
PRIMARY KEY("model_id","backend_pair")
45+
)'''
46+
CREATE_MAP = '''CREATE TABLE "localization_map" (
47+
"incons_id" INTEGER NOT NULL,
48+
"layer_name" TEXT NOT NULL,
49+
"outputs_delta" REAL,
50+
"outputs_R" REAL,
51+
"gradients_delta" REAL,
52+
"gradients_R" REAL,
53+
"weights_delta" REAL,
54+
"inbound_layers" TEXT,
55+
PRIMARY KEY("incons_id","layer_name")
56+
)'''
57+
58+
conn.execute(CLEAR_MODEL)
59+
conn.execute(CLEAR_INCONS)
60+
conn.execute(CLEAR_MAP)
61+
conn.execute(CREATE_MODEL)
62+
conn.execute(CREATE_INCONS)
63+
conn.execute(CREATE_MAP)
64+
conn.commit()
65+
66+
print("Done!")

0 commit comments

Comments
 (0)