Skip to content

Commit 3c74e8b

Browse files
themattinthehattfarzad-ziaieSelmaan
authored
support for multiple data streams (#187)
* Multi view (#115) * [add] multiview dictionary and multiviewdataset * [add] multiview to get_dataset * [fix] copy package & type for dataset * [add] dynamic keys * [add] added Multiviewheatmaplabeled to base.basesupervisedtracker type * [add] mouse mirror dataset * [add] test dataset for Multiviewheatmap * [add] test for multiview heatmap data module * [add] hydra raise notinplementederror for regression and multiview * [add] prediction handler and keypoints structure for Multiview * [fix] keypoints for multiview in prediction is now saved in csvs correctly * [add] make_multiview_dataset * [fix] datacheck is more strict now * [change] the dataset is now stacked along channel dimension. [fix] tests are vaild now * [fix] images are stacked along the batch dimension and representation was reshaped * [add] data type to fusion * [fix] datadicts in data utils * [add] test_supervised_multiview_heatmap and skipped video_dataloader * update code to use original image and video pixel coordinates for all predictions (#117) * Squashed commit of the following: commit 9b7de3c Author: Selmaan <sc4551@columbia.edu> Date: Mon Sep 18 16:39:04 2023 -0400 Update config_birdCOM.yaml commit 1aae9de Author: Selmaan <sc4551@columbia.edu> Date: Mon Sep 18 15:59:43 2023 -0400 update configs for detector net Added params to default config need for dynamic crop algorithm and detector network. Then created a new bird config with edited fields * add bbox to dataset dictionaries add a xyhw bbox entry to dictionary for dataset results, for example and batch. Also update the base dataset class (for backwards compatability) to output a bounding box in the __getitem__ call. This 'default' bbox is just the full image dimensions * Update datasets.py fix automatic detection of image height and width * Update heatmap_tracker.py add code to convert from bounding box to original image dimension coordinates * Update test_pca.py fix keys in unit test * add bbox conversion get_loss_inputs_labeled and predict_step now call convert_bbox_coords, which converts predicted keypoints from (potentially cropped) image intrinsic coordinates to original image coordinates, using bbox info * code linting fixed code formatting following flake8 * Update heatmap_tracker.py add label reminding me to implement bbox conversion later for unlabeled data * Update augmentations.py do not resize data in the augmentation pipeline when using dynamic crop algorithm (cropping will be handled by dynamic pipeline) * Update config_birdCOM.yaml * Update scripts.py * Update .gitignore * Update datasets.py add new detector dataset, and correct previous image height and width calculation in BastTrackingDataset * Update heatmap_tracker.py transform target along with predicted keypoints from transformed image coordinates to original image coordinates * Update scripts.py fix typo in get_detector_model * Update config_birdCOM.yaml set useful parameter values for this dataset * add dynamic labeled dicts these are unused for now, will be implemented in the future for multi-instance detection * Update config_birdCOM.yaml redo COM config to be independent pipeline from later POS pipeline * Update scripts.py remove image size checking when setting up dataset. unrelated, also set 'columns for singleview pca' to None automatically if it's not set in the config * remove keypoints rescaling in predict remove rescaling of keypoints according to static config info (which is now removed). Keypoints are dynamically rescaled in model's predict_step now already! * add bbox conversion for unlabeled data update predict step and get unsupervised losses to convert keypoints predicted on unlabeled video frame data to original image coords using bbox info * update dali dataloader with frame sizes dali dataloader now outputs the size of loaded video frames in bbox info * pre computing heatmaps is very slow I think there is very little upside to having these precomputed? Creates a big lag for my use case whenever I try to create a dataset. * add bbox to DynamicDict * image_orig_dims no longer in configs removed image_orig_dims from config, so also does not need to be copied here. It will always be inferred from video/image data during inference * Update config_birdCOM.yaml * Create config_birdCOM_backup.yaml * delete commented out code * code linting * add bbox to unlabeled batch keys * changes from pull request review mostly moving convert_bbox_coords to be a function in models.base rather than a method of heatmap tracker class * fix bbox to device with images fixes edge case where frames and bbox where on separate devices * add bbox conversion unit test the test artifically crops and shifts an image and offsets the detected keypoint locations accordingly, and verifies that bbox_conversion for the original and re-cropped data match * combine multiview and dynamic crop PRs * [add] compute_metrics for multiview, it is a loop for each file (#119) * bug fix for multi-view metric computation (#121) * final bug fixes for multiview * bbox bug fix; closes #109, #120 * [docs] multiview separate * Multiview (#126) * [fix] preds_file typo to list * [fix] list of preds_file is being processed now * [add] hydra for compute metrics1 * [add] multiview heatmap context * [add] multiview heatmap context conftest * [add] dataset test * [add] mview data module test * [add] context and thir tests * [add] dynamic naming dataset basic * [fix] flake8 * PR fixes * add bbox coord transform to context models --------- Co-authored-by: themattinthehatt <themattinthehatt@gmail.com> * tweaks to streamlit to show labeled data results from all views * [fix] fiftyone app now compatible with multiple views * [fix] dataset typechecking error, new unit tests * remove detector code from multiview branch * remove detector code from multiview branch * update IO code to properly find multiview videos * update video_pipe to work for multiple views * update LitDaliWrapper to work for multiple views * semisupervised multiview training without error * bug fixes with dali augmentations * multiview semisupervised context dataloader + model tests passing * affine transform bug fix + refactoring + unit test --------- Co-authored-by: Farzad Ziaie Nezhad <farzadziaien@gmail.com> Co-authored-by: Selmaan <sc4551@columbia.edu>
1 parent 4967266 commit 3c74e8b

File tree

54 files changed

+2832
-603
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+2832
-603
lines changed

‎.gitignore

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,3 @@
1-
lightning_logs/
2-
grid_artifacts/
3-
.vscode/
4-
outputs/
5-
multirun/
6-
preds/
7-
scripts/.ipynb_checkpoints
8-
#scripts/configs_*
9-
101
# Byte-compiled / optimized / DLL files
112
__pycache__/
123
*.py[cod]
@@ -144,10 +135,19 @@ dmypy.json
144135
**/.DS_Store
145136

146137
# specific stuff like csv predictions on test_vid.mp4
147-
toy_datasets/toymouseRunningData/unlabeled_videos/*.csv
148-
toy_datasets/toymouseRunningData/unlabeled_videos/test_vid_labeled.mp4
149-
toy_datasets/toymouseRunningData/unlabeled_videos/test_vid_*.mp4
150-
toy_datasets/toymouseRunningData/barObstacleScaling1/*.npy
138+
data/mirror-mouse-example/videos/*.csv
139+
data/mirror-mouse-example/videos/test_vid_*.mp4
151140

152-
# split dataset that is computed on the fly
141+
# other datasets
153142
data/mirror-mouse-example_split/
143+
data/Chickadee
144+
145+
# random other outputs that might end up in the repo
146+
lightning_logs/
147+
grid_artifacts/
148+
.vscode/
149+
outputs/
150+
multirun/
151+
preds/
152+
scripts/.ipynb_checkpoints
153+
tb_logs

‎docs/api/lightning_pose.data.datasets.BaseTrackingDataset.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,15 @@ BaseTrackingDataset
55

66
.. autoclass:: BaseTrackingDataset
77
:show-inheritance:
8+
9+
.. rubric:: Attributes Summary
10+
11+
.. autosummary::
12+
13+
~BaseTrackingDataset.height
14+
~BaseTrackingDataset.width
15+
16+
.. rubric:: Attributes Documentation
17+
18+
.. autoattribute:: height
19+
.. autoattribute:: width

‎docs/api/lightning_pose.data.datasets.HeatmapDataset.rst

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,25 @@ HeatmapDataset
55

66
.. autoclass:: HeatmapDataset
77
:show-inheritance:
8+
9+
.. rubric:: Attributes Summary
10+
11+
.. autosummary::
12+
13+
~HeatmapDataset.output_shape
14+
15+
.. rubric:: Methods Summary
16+
17+
.. autosummary::
18+
19+
~HeatmapDataset.compute_heatmap
20+
~HeatmapDataset.compute_heatmaps
21+
22+
.. rubric:: Attributes Documentation
23+
24+
.. autoattribute:: output_shape
25+
26+
.. rubric:: Methods Documentation
27+
28+
.. automethod:: compute_heatmap
29+
.. automethod:: compute_heatmaps

‎docs/api/lightning_pose.data.datasets.MultiviewHeatmapDataset.rst

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
MultiviewHeatmapDataset
2+
=======================
3+
4+
.. currentmodule:: lightning_pose.data.datasets
5+
6+
.. autoclass:: MultiviewHeatmapDataset
7+
:show-inheritance:
8+
9+
.. rubric:: Attributes Summary
10+
11+
.. autosummary::
12+
13+
~MultiviewHeatmapDataset.height
14+
~MultiviewHeatmapDataset.num_views
15+
~MultiviewHeatmapDataset.output_shape
16+
~MultiviewHeatmapDataset.width
17+
18+
.. rubric:: Methods Summary
19+
20+
.. autosummary::
21+
22+
~MultiviewHeatmapDataset.check_data_images_names
23+
~MultiviewHeatmapDataset.fusion
24+
25+
.. rubric:: Attributes Documentation
26+
27+
.. autoattribute:: height
28+
.. autoattribute:: num_views
29+
.. autoattribute:: output_shape
30+
.. autoattribute:: width
31+
32+
.. rubric:: Methods Documentation
33+
34+
.. automethod:: check_data_images_names
35+
.. automethod:: fusion

‎docs/api/lightning_pose.data.utils.MultiviewHeatmapLabeledBatchDict.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
MultiviewHeatmapLabeledBatchDict
2+
================================
3+
4+
.. currentmodule:: lightning_pose.data.utils
5+
6+
.. autoclass:: MultiviewHeatmapLabeledBatchDict
7+
:show-inheritance:

‎docs/api/lightning_pose.data.utils.MultiviewHeatmapLabeledExampleDict.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
MultiviewHeatmapLabeledExampleDict
2+
==================================
3+
4+
.. currentmodule:: lightning_pose.data.utils
5+
6+
.. autoclass:: MultiviewHeatmapLabeledExampleDict
7+
:show-inheritance:

‎docs/api/lightning_pose.data.utils.MultiviewLabeledBatchDict.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
MultiviewLabeledBatchDict
2+
=========================
3+
4+
.. currentmodule:: lightning_pose.data.utils
5+
6+
.. autoclass:: MultiviewLabeledBatchDict
7+
:show-inheritance:

‎docs/api/lightning_pose.data.utils.MultiviewLabeledExampleDict.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
MultiviewLabeledExampleDict
2+
===========================
3+
4+
.. currentmodule:: lightning_pose.data.utils
5+
6+
.. autoclass:: MultiviewLabeledExampleDict
7+
:show-inheritance:

‎docs/api/lightning_pose.data.utils.MultiviewUnlabeledBatchDict.rst

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
MultiviewUnlabeledBatchDict
2+
===========================
3+
4+
.. currentmodule:: lightning_pose.data.utils
5+
6+
.. autoclass:: MultiviewUnlabeledBatchDict
7+
:show-inheritance:
8+
9+
.. rubric:: Attributes Summary
10+
11+
.. autosummary::
12+
13+
~MultiviewUnlabeledBatchDict.is_multiview
14+
15+
.. rubric:: Attributes Documentation
16+
17+
.. autoattribute:: is_multiview

‎docs/api/lightning_pose.data.utils.UnlabeledBatchDict.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,13 @@ UnlabeledBatchDict
55

66
.. autoclass:: UnlabeledBatchDict
77
:show-inheritance:
8+
9+
.. rubric:: Attributes Summary
10+
11+
.. autosummary::
12+
13+
~UnlabeledBatchDict.is_multiview
14+
15+
.. rubric:: Attributes Documentation
16+
17+
.. autoattribute:: is_multiview

0 commit comments

Comments
 (0)