Visual Monitoring of Crop Production Systems: A multimodal approach

The world faces a growing risk of food shortages due to climate change and the increasing global population. Digital transformation of agriculture holds the potential for creating efficient, resilient and eco-friendly crop production systems. In recent years, automatic monitoring methods based on different imaging technologies (e.g., hyperspectral, thermal imaging) have been successfully applied to identify drought in plants, detect decay in fruits and other tasks. As such, they have proven to be valuable tools for achieving sustainability in agricultural systems. Established methods, however, mostly rely on using a single imaging technology at once. This neglects the potential of complementary information that could be obtained with multiple cameras and could lead to improved monitoring accuracy. Thus, the goal of this dissertation project is to utilize multiple camera technologies simultaneously for the monitoring of crop production processes and to develop powerful multimodal machine learning techniques to precisely predict relevant target traits. In this project, the input modalities are represented by different imaging technologies (e.g., thermal camera for temperature, hyperspectral camera for waveband information (incl. visible light), lidar camera for depth information). To demonstrate that the developed machine learning methods are generalizable, they will be evaluated in three real-world use cases, which represent the entire spectrum of the “One Health” concept: 1) health/fertility assessment of soil, 2) detection of nutrient deficiency in plants, 3) early detection of decay in fruits and vegetables. Contributions of this thesis will be 1) the design of novel multimodal machine learning techniques for the prediction of biological traits from multiple camera modalities based on deep learning methodology, 2) the development of modality fusion methods that enable robustness to missing modalities and allow for the joint prediction of multiple traits, 3) the detailed evaluation of practical feasibility and level of improvement over unimodal methods in real-world use cases. With the developed methods, this dissertation aims to make a contribution to a more secure global food supply.