Dynamic Multimodal Prompt Deep Collaboration Strategy For Classification and Question-Answering Problem in Industrial Training

Traditional training methods involve learning through books or e-books, followed by practical training, which is inefficient. To address the issue of low efficiency in traditional training methods, this paper proposes an innovative industrial training approach based on multimodal large model technology. By constructing a Chinese multimodal industrial dataset and applying a novel fine-tuning approach, we developed a multimodal large model capable of understanding industrial knowledge. The experimental results show that under conditions of limited training epochs, our method achieves performance comparable to that of methods with higher training epochs, demonstrating its effectiveness under resource-constrained settings. The main contributions include: 1) the creation of the first Chinese industrial parts multimodal dataset generated by a hybrid multimodal data generation method; 2) the proposal of a dynamic multimodal prompt deep collaboration strategy, which greatly improves the model’s performance in classification and generative understanding tasks; and 3) The experimental validation demonstrates that our method achieves outstanding results across various experiments and exhibits significant performance advantages. Our dataset is available at https://dx.doi.org/10.21227/3ngv-br47