Local visual feature fusion via maximum margin multimodal deep neural network

In this letter, we consider improving the image categorization performance by exploiting multiple local descriptors on the image. To achieve this goal, a novel deep learning configuration called maximum margin multimodal deep neural network (3mDNN) is proposed to learn joint feature from different data views. The local feature representations encoded by 3mDNN exhibit two significant advantages: (1) involving the information of multiple descriptors and (2) exhibiting discriminative ability. The whole deep architecture is well solved by the typical back propagation (BP) method and its performances are verified on three benchmark image datasets.