FruitNeRF: A Unified Neural Radiance Field based Fruit Counting Framework

Meyer L, Gilson A, Schmid U, Stamminger M (2024)

Publication Type: Conference contribution

Publication year: 2024

Journal

IEEE/RSJ International Conference on Intelligent Robots and Systems IEEE

Publisher: Institute of Electrical and Electronics Engineers Inc.

Pages Range: 1-8

Conference Proceedings Title: IEEE International Conference on Intelligent Robots and Systems

Event location: Abu Dhabi AE

ISBN: 9798350377705

DOI: 10.1109/IROS58592.2024.10802065

Abstract

We introduce FruitNeRF, a unified novel fruit counting framework that leverages state-of-the-art view synthesis methods to count any fruit type directly in 3D. Our framework takes an unordered set of posed images captured by a monocular camera and segments fruit in each image. To make our system independent of the fruit type, we employ a foundation model that generates binary segmentation masks for any fruit. Utilizing both modalities, RGB and semantic, we train a semantic neural radiance field. Through uniform volume sampling of the implicit Fruit Field, we obtain fruit-only point clouds. By applying cascaded clustering on the extracted point cloud, our approach achieves precise fruit count. The use of neural radiance fields provides significant advantages over conventional methods such as object tracking or optical flow, as the counting itself is lifted into 3D. Our method prevents double counting fruit and avoids counting irrelevant fruit. We evaluate our methodology using both real-world and synthetic datasets. The real-world dataset consists of three apple trees with manually counted ground truths, a benchmark apple dataset with one row and ground truth fruit location, while the synthetic dataset comprises various fruit types including apple, plum, lemon, pear, peach, and mango. Additionally, we assess the performance of fruit counting using the foundation model compared to a U-Net.

Authors with CRIS profile

Lukas Meyer Chair of Visual Computing (LGDV) Marc Stamminger Chair of Visual Computing (LGDV)

Involved external institutions

Fraunhofer-Institut für Integrierte Schaltungen (IIS) DE Germany (DE) Otto-Friedrich-Universität Bamberg DE Germany (DE)

How to cite

APA:

Meyer, L., Gilson, A., Schmid, U., & Stamminger, M. (2024). FruitNeRF: A Unified Neural Radiance Field based Fruit Counting Framework. In IEEE International Conference on Intelligent Robots and Systems (pp. 1-8). Abu Dhabi, AE: Institute of Electrical and Electronics Engineers Inc..

MLA:

Meyer, Lukas, et al. "FruitNeRF: A Unified Neural Radiance Field based Fruit Counting Framework." Proceedings of the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2024, Abu Dhabi Institute of Electrical and Electronics Engineers Inc., 2024. 1-8.

BibTeX: Download