Enhancing In-Field Robotic Blueberry Fruit Phenotyping: Customized BerryNet Through Learning Pixel-wise Labels from Segment Anything Model (SAM)
The repository is for the paper: Enhancing In-Field Robotic Blueberry Fruit Phenotyping: Customized BerryNet Through Learning Pixel-wise Labels from Segment Anything Model (SAM), including the code and dataset for reproducing.
Fig. 1: Diagram of the proposed blueberry fruit phenotyping workflow involving four stages: data collection, training dataset generation, model training, and phenotyping traits extraction.
Figure 2. Illustration of the proposed automated pixel-wise label generation process for blueberry fruits at different maturiety stages. (a) Bounding boxes from a previous detection dataset (Z. Li et al., 2023); (b) Bounding boxes re-classified into three categories: immature (yellow), semi-mature(red), and mature (blue), using a maturity classifier; (c) Pixel-wise mask labels generated using the Segment Anything Model.
Figure 3. Illustration of the BerryNet framework. It incorporated three major enhancements: 1) enhancing P2 layer to better capture features of small objects; 2) implementing BiFPN for improved feature fusion, and 3) replacing C2f block with the more efficient C2f-faster block to accelerate inference.
pip install ultralytics
pip install git+https://github.com/facebookresearch/segment-anything.git
Clone the repository to local machine:
git clone https://github.com/UGA-BSAIL/BerryNet.git
Create a virtual env and Install the required packages :
conda create -n BerryNet python=3.8
conda activate BerryNet
pip install ultralytics
pip install git+https://github.com/facebookresearch/segment-anything.git
We modified the original YOLOv8 repository for more module support (yolov8-BerryNet\ultralytics\nn\extra_modules). For letting ultralytics point to the modified repository,
pip uninstall ultralytics
This paper released four datasets for comprhensive research of blueberry, which are availiable on kaggle:
- Blueberry Fruit Detection
- Blueberry Maturity Classification
- Blueberry Pixel-wise Segmentation
- Blueberry Cluster Detection
This method requred the detection dataset as the initial prompt, and weight of maturity classifier and SAM for inference. MOdified the path of the dataset and the model in the script of SAM-based-labeling.py:
python script\pixle-wise_labeling.py
Parameters:
- image_folder = '/path/to/image_folder'
- annotation_folder = '/path/to/annotation_folder'
- checkpoint = torch.load('/path/to/best_model.pth', map_location=torch.device('cpu')) # path to the maturity classifier
- sam_checkpoint = "/path/to/sam_vit_h_4b8939.pth" # path to the SAM model
BerryNet's architecture was defined in yolov8-BerryNet\ultralytics\models\v8\yolov8-c2f_faster-p2-bifpn-seg.yaml. For training the model, run the script of train_blueberry_c2f-faster_p2_bifpn_seg_640.py (larger model select the 1280) under the path of yolov8-BerryNet folder:
cd yolov8-BerryNet
python train_blueberry_c2f-faster_p2_bifpn_seg_640.py
Before running the script, please modify the path of the dataset and the model configuration file in the script. You can try more yaml files for different model architecture.
The pre-trained models are available at weight.
- Maturity_classifier:
- Segment Anything Model:
- Cluster Detection:
- Fruit Segmentaiton:
For model inference, run the script of BerryNet_phenotyping_extraction_split.py under the script folder:
python script\BerryNet_phenotyping_extraction_split.py
Parameters:
- model_path = " " # path to the BerryNet model
- image_folder = " " # path to the image folder
- save_path = " " # path to the save folder
If you find this work or code useful, please cite:
@article{li2025field,
title={In-field blueberry fruit phenotyping with a MARS-PhenoBot and customized BerryNet},
author={Li, Zhengkun and Xu, Rui and Li, Changying and Munoz, Patricio and Takeda, Fumiomi and Leme, Bruno},
journal={Computers and Electronics in Agriculture},
volume={232},
pages={110057},
year={2025},
publisher={Elsevier}
}
@inproceedings{li2023blueberry,
title={Blueberry Yield Estimation Through Multi-View Imagery with YOLOv8 Object Detection},
author={Li, Zhengkun and Li, Changying and Munoz, Patricio},
booktitle={2023 ASABE Annual International Meeting},
pages={1},
year={2023},
organization={American Society of Agricultural and Biological Engineers}
}