Skip to main navigation Skip to search Skip to main content

A fine-grained performance bottleneck analysis method for HDFS

  • Beihang University

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

The performance issue of HDFS has always been a great concern due to its widely adoption in both production and research environments. However, a fine-grained performance analysis tool is missing to effectively identify the bottlenecks as well as to provide useful guidance for performance optimization. In this paper, we propose a fine-grained performance bottleneck analysis tool, which extends HTrace with fine-grained instrumentation points that are missing in Hadoop official distribution. In addition, we propose an effective trace merging method that improves the understandability of our analysis. We analyze the performance of HDFS under different kinds of workloads and get undiscovered insights.

Original languageEnglish
Title of host publicationNetwork and Parallel Computing - 15th IFIP WG 10.3 International Conference, NPC 2018, Proceedings
EditorsMarc Snir, Mateo Valero, Feng Zhang, Hironori Kasahara, Jidong Zhai, Hai Jin
PublisherSpringer Verlag
Pages159-163
Number of pages5
ISBN (Print)9783030056766
DOIs
StatePublished - 2018
Event15th IFIP International Conference on Network and Parallel Computing, NPC 2018 - Muroran, Japan
Duration: 29 Nov 20181 Dec 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11276 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference15th IFIP International Conference on Network and Parallel Computing, NPC 2018
Country/TerritoryJapan
CityMuroran
Period29/11/181/12/18

Keywords

  • Bottleneck analysis
  • HDFS
  • Instrumentation
  • Performance optimization

Fingerprint

Dive into the research topics of 'A fine-grained performance bottleneck analysis method for HDFS'. Together they form a unique fingerprint.

Cite this