Plant Vision

Plants are the major organisms that can absorb the light energy from the sun to produce biomass and oxygen. One key problem in studying plant growth is to understand the photosynthetic activities of plants under various external stimuli or genetic variations. Because leaves at different developmental ages may response to the change of the environmental conditions and gene mutations in very different ways, it is important to conduct a leaf-level analysis of the photosynthetic efficiency. Motivated by the needs in plant biology, given images and videos captured by visual sensors, the goal of the plant vision project is to develop advanced computer vision algorithms to automatically, accurately and efficiently estimate the structure of a plant, which includes 2D multi-leaf alignment and tracking, and 3D reconstruction of all leaves in a plant.

We have developed a framework based on the well-known Chamfer Matching algorithm. The input to our system is a fluorescence image or video of a plant, which is captured in a growth chamber (Fig. 1). Multi-leaf alignment aims to segment/align all leaves with pre-defined leaf templates and estimate the two tip points of each leaf. Multi-leaf tracking aims to track all leaves over time based on the alignment results of one frame. The leaf alignment and tracking results can directly benefit the study of leaf behavior in plant biology, such as leaf growth, leaf-level photosynthesis, leaf-level variations in plant mutant, etc. We have recently extended our system to process RGB videos of plants as well.

Fig. 1 Overview of leaf alignment and tracking system

Multi-leaf Alignment

As shown in Fig. 2, the multi-leaf tracking algorithm consists of two steps. Firstly, a set of templates are applied to the test image to generate the same amount of leaf candidates. Secondly, we develop a multi-objective optimization process to select a subset of leaf candidates. The objective is to select a minimal number of leaf candidates with smaller Chamfer distances to cover the test image mask as much as possible.

Fig. 2 Overview of multi-leaf alignment

Multi-leaf Tracking

Multi-leaf tracking is an extension of the leaf alignment algorithm. Given a fluorescence plant video taken over time, we first apply the alignment algorithm to the last frame of the video, and then continuously apply template transformation to the current leaf candidates in order to fit to the previous frame. We develop an objective function considering the Chamfer Matching distances, test image mask, and the rotation angels of all leaves. An example of multi-leaf tracking is shown below.

Data Set