Design of a Multimodal Explainable AI Framework for Sustainable Intelligent Agriculture: Methodology and Expected Outcomes

Agricultural sustainability is a major concern currently in globe, as it is related to the food security and scarcity. There is always a need for multimodal and explainable frameworks for achieving the sustainability in the field of agriculture. The paper proposes a framework consisting of the integration of different modalities with explainable insights of the detection and prediction decision of crop management and precision agriculture.

A popular multimodal dataset Pheno4D is utilized for this work. The different modalities including RGB images, thermal images, 3D point cloud and time series data. These modalities offer a fair generalization of the proposed framework. The framework consists of a hybrid fusion architecture and attention-based model. Explainability and interpretability are achieved via attention mechanisms and post-hoc networks utilizing the frameworks including Shapley values (SHAP) and Grad-CAM. The proposed model primarily offers the disease prediction and its severity. The methodology could be extended for prediction of crop yield, and optimization of water-resources with maintaining the trust and interpretability.