RUPredHadoop: Resources utilization predictor for hadoop with large-scale clusters

Shangming Ning; Fei Teng; Yunshu Li; Zhe Cui; Lei Yu; Shengdong Du

doi:10.1007/978-981-13-2922-7_32

RUPredHadoop: Resources utilization predictor for hadoop with large-scale clusters

Shangming Ning
, Fei Teng^*
, Yunshu Li
, Zhe Cui
, Lei Yu
, Shengdong Du

^*Corresponding author for this work

Sino-French Engineer School/School of General Engineering

Southwest Jiaotong University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Apache Hadoop is a widely used distributed system in large-scale production environment. With the increasing size of data volume and cluster scale, its performance is limited by inappropriate resources utilization. This paper introduces a resources utilization predictor (RUPredHadoop) to predict utilization of cpu, memory, read/write rate of disk and network, especially for large-scale Hadoop clusters. In terms of the similarity of data and workflow in Hadoop, the pattern of resource utilization for a single task is proposed, and then formulized by a single task model. Besides that, the distribution of fine-grained runtime is studied, so that a parallel-batch-tasks-based model could regenerate the whole Mapreduce job by migrating the single task model from the minimum cluster to a large-scale production cluster. With RUPredHadoop, we can locate the resource bottleneck for Hadoop clusters, meanwhile we can agilely configure clusters for applications with massive data. The performance of RUPredHadoop is validated by a test cluster with 35 nodes and a production cluster with 80 nodes. Results show that the normalization error is below 10% for benchmark applications with maximum 100 TB data.

Original language	English
Title of host publication	Big Data - 6th CCF Conference, Big Data 2018, Proceedings
Editors	Zongben Xu, Jiajun Bu, Yunquan Zhang, Xinbo Gao, Qiguang Miao
Publisher	Springer Verlag
Pages	477-493
Number of pages	17
ISBN (Print)	9789811329210
DOIs	https://doi.org/10.1007/978-981-13-2922-7_32
State	Published - 2018
Event	6th CCF Academic Conference on Big Data, CCF Big Data 2018 - Xi'an, China Duration: 11 Oct 2018 → 13 Oct 2018

Publication series

Name	Communications in Computer and Information Science
Volume	945
ISSN (Print)	1865-0929

Conference

Conference	6th CCF Academic Conference on Big Data, CCF Big Data 2018
Country/Territory	China
City	Xi'an
Period	11/10/18 → 13/10/18

Keywords

Data center
Hadoop
Performances
Predictor

Access to Document

10.1007/978-981-13-2922-7_32

Cite this

Ning, S., Teng, F., Li, Y., Cui, Z., Yu, L., & Du, S. (2018). RUPredHadoop: Resources utilization predictor for hadoop with large-scale clusters. In Z. Xu, J. Bu, Y. Zhang, X. Gao, & Q. Miao (Eds.), Big Data - 6th CCF Conference, Big Data 2018, Proceedings (pp. 477-493). (Communications in Computer and Information Science; Vol. 945). Springer Verlag. https://doi.org/10.1007/978-981-13-2922-7_32

@inproceedings{78d7bc25cb48434aa1ceb3fa747bf51c,

title = "RUPredHadoop: Resources utilization predictor for hadoop with large-scale clusters",

abstract = "Apache Hadoop is a widely used distributed system in large-scale production environment. With the increasing size of data volume and cluster scale, its performance is limited by inappropriate resources utilization. This paper introduces a resources utilization predictor (RUPredHadoop) to predict utilization of cpu, memory, read/write rate of disk and network, especially for large-scale Hadoop clusters. In terms of the similarity of data and workflow in Hadoop, the pattern of resource utilization for a single task is proposed, and then formulized by a single task model. Besides that, the distribution of fine-grained runtime is studied, so that a parallel-batch-tasks-based model could regenerate the whole Mapreduce job by migrating the single task model from the minimum cluster to a large-scale production cluster. With RUPredHadoop, we can locate the resource bottleneck for Hadoop clusters, meanwhile we can agilely configure clusters for applications with massive data. The performance of RUPredHadoop is validated by a test cluster with 35 nodes and a production cluster with 80 nodes. Results show that the normalization error is below 10\% for benchmark applications with maximum 100 TB data.",

keywords = "Data center, Hadoop, Performances, Predictor",

author = "Shangming Ning and Fei Teng and Yunshu Li and Zhe Cui and Lei Yu and Shengdong Du",

note = "Publisher Copyright: {\textcopyright} Springer Nature Singapore Pte Ltd. 2018.; 6th CCF Academic Conference on Big Data, CCF Big Data 2018 ; Conference date: 11-10-2018 Through 13-10-2018",

year = "2018",

doi = "10.1007/978-981-13-2922-7\_32",

language = "英语",

isbn = "9789811329210",

series = "Communications in Computer and Information Science",

publisher = "Springer Verlag",

pages = "477--493",

editor = "Zongben Xu and Jiajun Bu and Yunquan Zhang and Xinbo Gao and Qiguang Miao",

booktitle = "Big Data - 6th CCF Conference, Big Data 2018, Proceedings",

address = "德国",

}

Ning, S, Teng, F, Li, Y, Cui, Z, Yu, L & Du, S 2018, RUPredHadoop: Resources utilization predictor for hadoop with large-scale clusters. in Z Xu, J Bu, Y Zhang, X Gao & Q Miao (eds), Big Data - 6th CCF Conference, Big Data 2018, Proceedings. Communications in Computer and Information Science, vol. 945, Springer Verlag, pp. 477-493, 6th CCF Academic Conference on Big Data, CCF Big Data 2018, Xi'an, China, 11/10/18. https://doi.org/10.1007/978-981-13-2922-7_32

RUPredHadoop: Resources utilization predictor for hadoop with large-scale clusters. / Ning, Shangming; Teng, Fei; Li, Yunshu et al.
Big Data - 6th CCF Conference, Big Data 2018, Proceedings. ed. / Zongben Xu; Jiajun Bu; Yunquan Zhang; Xinbo Gao; Qiguang Miao. Springer Verlag, 2018. p. 477-493 (Communications in Computer and Information Science; Vol. 945).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - RUPredHadoop

T2 - 6th CCF Academic Conference on Big Data, CCF Big Data 2018

AU - Ning, Shangming

AU - Teng, Fei

AU - Li, Yunshu

AU - Cui, Zhe

AU - Yu, Lei

AU - Du, Shengdong

N1 - Publisher Copyright: © Springer Nature Singapore Pte Ltd. 2018.

PY - 2018

Y1 - 2018

N2 - Apache Hadoop is a widely used distributed system in large-scale production environment. With the increasing size of data volume and cluster scale, its performance is limited by inappropriate resources utilization. This paper introduces a resources utilization predictor (RUPredHadoop) to predict utilization of cpu, memory, read/write rate of disk and network, especially for large-scale Hadoop clusters. In terms of the similarity of data and workflow in Hadoop, the pattern of resource utilization for a single task is proposed, and then formulized by a single task model. Besides that, the distribution of fine-grained runtime is studied, so that a parallel-batch-tasks-based model could regenerate the whole Mapreduce job by migrating the single task model from the minimum cluster to a large-scale production cluster. With RUPredHadoop, we can locate the resource bottleneck for Hadoop clusters, meanwhile we can agilely configure clusters for applications with massive data. The performance of RUPredHadoop is validated by a test cluster with 35 nodes and a production cluster with 80 nodes. Results show that the normalization error is below 10% for benchmark applications with maximum 100 TB data.

AB - Apache Hadoop is a widely used distributed system in large-scale production environment. With the increasing size of data volume and cluster scale, its performance is limited by inappropriate resources utilization. This paper introduces a resources utilization predictor (RUPredHadoop) to predict utilization of cpu, memory, read/write rate of disk and network, especially for large-scale Hadoop clusters. In terms of the similarity of data and workflow in Hadoop, the pattern of resource utilization for a single task is proposed, and then formulized by a single task model. Besides that, the distribution of fine-grained runtime is studied, so that a parallel-batch-tasks-based model could regenerate the whole Mapreduce job by migrating the single task model from the minimum cluster to a large-scale production cluster. With RUPredHadoop, we can locate the resource bottleneck for Hadoop clusters, meanwhile we can agilely configure clusters for applications with massive data. The performance of RUPredHadoop is validated by a test cluster with 35 nodes and a production cluster with 80 nodes. Results show that the normalization error is below 10% for benchmark applications with maximum 100 TB data.

KW - Data center

KW - Hadoop

KW - Performances

KW - Predictor

UR - https://www.scopus.com/pages/publications/85055676629

U2 - 10.1007/978-981-13-2922-7_32

DO - 10.1007/978-981-13-2922-7_32

M3 - 会议稿件

AN - SCOPUS:85055676629

SN - 9789811329210

T3 - Communications in Computer and Information Science

SP - 477

EP - 493

BT - Big Data - 6th CCF Conference, Big Data 2018, Proceedings

A2 - Xu, Zongben

A2 - Bu, Jiajun

A2 - Zhang, Yunquan

A2 - Gao, Xinbo

A2 - Miao, Qiguang

PB - Springer Verlag

Y2 - 11 October 2018 through 13 October 2018

ER -

RUPredHadoop: Resources utilization predictor for hadoop with large-scale clusters

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this