DisITQ: A distributed iterative quantization hashing learning algorithm

In the field of big data retrieval, hashing based approxi-mate nearest neighbors (ANN) search has attracted many at-tentions. However, most existing hashing algorithms are learned from the centralized settings and based on small scale datasets, or in other words, they are single machine ap-proaches which load the training data into memory to get models. For big data processing, models learned from large scale datasets which have the properties of big data such as variety often have better performance. However, there are two critical problems when training datasets are in very large size. First, a single compute node can't load all the data into memory to train hashing models. Second, in real-word appli-cations, the data is often stored or even collected in a distrib-uted manner, and it's infeasible to gather all data into a fu-sion center because of the prohibitively expensive communi-cation and computation overhead. In this article, we present a distributed learning algorithm which is based on MapReduce and Iterative Quantization (ITQ) to train hashing functions. The proposed method, named as distributed iterative quanti-zation hashing (DisITQ), can not only be performed on large scale datasets, but can also be applied to distributed data storing scenarios. Massive experiments carried out on large scale datasets demonstrate the time efficiency and the accu-racy advantages of the method we proposed in comparison with the state-of-the-art hashing algorithms.