面向海洋的多模态智能计算:挑战、进展和展望

The marine-oriented research is essential to high-quality of human-based development. But, the current recognition of the ocean system is less than 5%. To understand the ocean, big marine data is acquired from observation, monitoring, investigation and statistics. Thanks to the development of the multi-scaled ocean observation system, the extensive of multi-modal marine oriented data has developed via remote sensing image, spatio-temporal analysis, simulation data, literature review and video & audio monitoring. To resilient the sustainable development of human society, current deep analysis and multimodal ocean data mining method has promoted the marine understanding on the aspects of ocean dynamic processes, energy and material cycles, the evolution of blue life, scientific discovery, healthy environment, and the quick response of extreme weather and climate change. Compared to traditional big data, the multi-modal big ocean data has its unique features, such as the super-giant system (covering 71% of the earth’s surface, daily increment (10 TB), super multi-perspectives (“land-sea-air-ice-earth based” coupling, “hydrometeorological-acoustical-optical and electromagnetic-based” polymorphism), super spatial scale (“centimeter to hundreds kilometer based”), and temporal scale (“micro-second to inter-decadal based”). These features-derived challenges of existing multi-modal intelligent computing technology have to deal with such problems as cross-scale and multi-modal fusion analyses, multi-disciplinary and multi-domain coordinated reasoning, large computing power based multi-architecture compatible applications. We systematically introduce the cross-cutting researches of intelligent perception, cognition, and prediction for marine phenomena/processes based on multimodal data technology. First, we clarify the research objects, scientific problems, and typical application scenarios of marine multimodal intelligent computing through the evolution analysis of the lifecycle of marine science big data. Next, we target the differences between ocean data distribution and calculation patterns. We illustrate the uniqueness and scientific challenges of multimodal big marine data on the basis of modeling description, cross-modal correlation, inference prediction, and high-performance computing. 1) To bridge the “task gap” between big data and specific tasks for modeling description, we focus on effective feature extraction for related tasks of causality, differentiation, significance and robustness. The ocean-oriented differences and challenges are mainly discussed from six aspects including dynamic changes of physical structure, complex environmental noise, large intra-class differences, lack of reliable labels, unbalanced samples, and less public datasets. 2) To construct multi-circle layer, multi-scale and multi-perspective heterogeneous data, the cross-modal correlation modeling is obtained for reasonable integration of multi-model, effective reasoning of cross-model, and the multi-modalities of “heterogeneous gap bridging” through task matching, semantic consistency, and spatio-temporal correlation. The ocean field issue is mainly affected by four aspects of uneven data, large scale span, strong constraints of temporal and spatial, and high correlation of dimensions. 3) To fill the “unknown gap” of spatio-temporal information loss in the evolution of ocean, the reasoning and prediction requires the prior knowledge, experience, and reasoning ability in the field of modeling. The main differences of ocean fields are reflected in the three issues of dynamic evolution, spatiotemporal heterogeneity, and non-independent samples. 4) To reduce the “computing gap” between complex computing and real-time online analysis of marine super-giant systems, it is necessary to deal with the huge amount of data challenges in high-performance computing problems like increased resolution and the ocean processes refinement of online response analysis. In addition, we sort out and introduce existing work of typical application scenarios, such as marine multimedia content analysis, visual analysis, big data prediction, and high-performance computing. 1) Multimedia content analysis: we compare the technical features of existing marine research methods on the five aspects of target recognition, target re-identification, target retrieval, phenomenon/process recognition, and open datasets. 2) Visual analysis of marine big data: we summarize the matching issues of dynamic changes of physical structure, high correlation dimensions, and large-scale spans from the perspective of visualization, visualization analysis, and visualization system. 3) Ocean multimodal big data reasoning prediction: we review the existing research work from the perspectives of data-driven prediction and prediction of marine environmental variables, construction of marine knowledge graph, and knowledge reasoning. 4) High-performance computing issues of ocean multi-modal big data: we introduce and compare the relevant work on the three perspectives of memory-computing collaboration, multi-model acceleration, and giant system evaluation. Finally, we predict the ocean multimodal intelligent computing issues to be resolved and the future direction of it.