Semantic segmentation and object detection pipeline for analyzing images of field-grown Kura clover (Trifolium ambiguum L.) plants.
Authors
Create a new virtual environment using your favorite (venv, conda, pyenv, etc.) Install all dependencies
$ pip install -r requirements
You can directly train the models in your environment by calling them directly
$ python train_object_detection_model.py
$ python train_segmentation_model.py
The top k=5 best model runs (based on the validation loss) will be stored in checkpoints.
The Perennial Legumes Program at The Land Institute cultivated XXXX unique accessions from the USDA National Plant Germplasm System (NPGS) in the summer of 2017. Each plant was cultivated as a single plant plot. Plots were imaged with a Canon DLSR camera at least one time during the season. Clover plant were framed by a standard sampling quadrat constructed of 3/4" Schedule 40 PVC pipe with the following dimensions:
- OD (HxW) = 18"x18"
- ID (HxW) = 16.25"x16.25"
The annotated classes for object detection bounding boxes are:
{
"clover": 1,
"quadrat": 2,
"quadrat_corner": 3
}
For the semantic segmentation masks we have the following classes:
{
"soil": 0,
"quadrat": 1,
"clover": 3
}
The full dataset including the images, annotations, metadata, sampling strategy and training/validation splits are published in Zenodo and can be found at:
Meyering et al (2025). K1702 - Kura Clover (Trifolium ambiguum) USDA Accession Image Dataset
This project focuses on segmenting and analyzing Kura clover plants grown in the field. The goal is to develop computer vision models that can:
- Detect and segment clover plants, and sampling quadrats from the soil background.
- Detect and localize the corner PVC elbows on the sampling quadrat.
- Compute the homology matrix between the image and the ROI bounded by the corners of the PVC quadrats.
- Extract the kura clover contours from the standardized, prediction masks.
- Calculate accession specific shape metrics such as solidity, canopy density, number of connected components, and fractal dimensions.
- Rank accessions based on desired phenotypic traits and map to geographic origin.
- Breeding values are computed for each accession based on shape/density metrics, enabling selection based on compactness, vigor, or spread.
- Object Detection (Quadrat Corner Detection): We use an EfficientDet-based object detector to locate the four PVC quadrat corners using the Ross Wightman
effdetlibrary. https://github.com/rwightman/efficientdet-pytorch - Semantic Segmentation (Plant/Soil/Quadrat): We will experiment with several different model architectures to segment kura clover from soil and quadrat background using the
segmentation-models-pytorchhttps://github.com/qubvel-org/segmentation_models.pytorch.
Detected quadrat corners are used to perform a perspective transform, warping the image to a standardized top-down view to correct for skew and perspective distortion.
From the warped segmentation masks, we extract standardized shape and density features using:
- Shape descriptors (e.g., area, solidity, convexity)
- Connected components analysis,
- Family-wise density estimates.
- Breeding Value Computation
- Breeding values are computed for each accession based on shape/density metrics, enabling selection based on compactness, vigor, or spread.
Here is an example of one of the plants with a small but very dense canopy
In contrast, here is an example of a plant with several small, dense clustered canopies results from rhizomatous growth below the soil.
Finally, this accession exhibits uniformly sparse canopy, with most of the leaves growing at the margins of the plant.
- From the warped masks, we compute:
- Total clover area
- Convex hull area
- Solidity (area / convex hull area)
- Number of clumps / components
- Average component size
- Edge distance metrics
- These metrics are used to evaluate morphological traits relevant to clover breeding (e.g., canopy density, spread, compactness).
Ranking accessions by morphological traits Identifying breeding targets for canopy architecture Estimating family-wise genetic effects (e.g., BLUPs) Automating phenotyping pipelines for breeding trials
data/
├── processed/
├── images/
└── targets/
├── raw/
├── images/
└── mat_files/
├── bboxes.csv
├── data_split.csv
├── plant_status.csv
└── SAM_points.csv
logs/
metadata/
├── obj_det_class_map.json
└── segmentation_class_map.json
outputs/
├── object_detection/
└──segmentation /
scripts/
src/
├── utils/
├── collate_functions.py
└── loggers.py
├── callbacks.py
├── datasets.py
├── eval.py
├── metrics.py
├── models.py
├── trainer.py
└── transforms.py
.gitignore
LICENSE
README.md
requirements.txt
train_object_detection_model.py
train_segmentation_model.py
- Labelbox: Image annotation
- PyTorch: Model training
- Albumentations: Image augmentation
- OpenCV: Image warping and homography
- NumPy
- SciPy: Connected components and shape analysis
- Segment Anything (SAM): Prompt-based segmentation
- Kirillov et al. (2023). Segment Anything DOI:10.48550/arXiv.2304.02643
- Tan et al. (2019). EfficientDet: Scalable and Efficient Object Detection DOI:10.48550/arXiv.1911.09070
- Chen et al. (2017). Rethinking Atrous Convolution for Semantic Image Segmentation DOI:10.48550/arXiv.1706.05587
- Chen et al. (2018). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation DOI:10.48550/arXiv.1802.02611
- Xie et al. (2021). SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers DOI:10.48550/arXiv.2105.15203
This work is part of our Kura clover breeding efforts at The Land Institute. Many thanks to the field and lab teams who assisted in image collection and annotation.