Image Caption Method Combining Multi-angle with Multi-modality

Image caption generation technology has attracted the interest of researchers due to its wide application in practical applications. It involves two vital areas of artificial intelligence: image processing and natural language processing. The existing method is to predict the generation of the next word based on the image features and the words generated in the previous state. However, it ignores the important role of text information. In this paper, we propose an image caption generation method that combines multi-angle with multimodality. The model firstly uses the fusion features of the global and local images as input. Picture description of the first sentence is generated using the baseline encoding-decoding model. The image caption which is generated firstly is then input into the sentence encoding network to generate a semantic feature vector of the first sentence. Then, the local visual feature vector of the image and the semantic eigenvector of the first sentence which are two different modal features, are combined and input into the attention-based language generation model to generate the next sentence. This allows our model to generate multi-angle descriptions in a targeted manner.