Optimized common parameter set extraction by benchmarking applications on a big data platform

Jongyeop Kim; Abhilash Kancharla; Jongho Seol; Noh Jin Park; Nohpill Park

doi:10.1109/SNPD.2018.8441059

Optimized common parameter set extraction by benchmarking applications on a big data platform

Jongyeop Kim, Abhilash Kancharla, Jongho Seol, Noh Jin Park, Nohpill Park

Information Technology

Research output: Contribution to book or proceeding › Conference article › peer-review

Abstract

This research proposes the methodology to extract common configuration parameter set by applying multiple benchmark applications including TeraSort., TestDFSIO, and MrBench on the Hadoop Distributed File System. In the process of determining parameter set for each stage, one parameter and its associated values selected which is reduced system performance in terms of overall execution time difference are measured by multiple applications on a Hadoop cluster. The experimental results demonstrate the proposed extended greedy manner provide a feasible benchmark model for the multiple tasks. In this way, we have found several parameter value sets that can reduce the execution time by 27% of the values provided by Hadoop default.

Original language	English
Title of host publication	Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018
Editors	Ha Jin Hwang, Lizhi Cai, Gun Huck Yeom, Tokuro Matsuo, Haeng Kon Kim, Hyun Yeo, Chung Sun Hong, Naoki Fukuta, Takayuki Ito, Huaikou Miao
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Print)	9781538658895
DOIs	https://doi.org/10.1109/SNPD.2018.8441059
State	Published - Aug 20 2018
Event	19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018 - Busan, Korea, Republic of Duration: Jun 27 2018 → Jun 29 2018

Publication series

Name	Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018

Conference

Conference	19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018
Country/Territory	Korea, Republic of
City	Busan
Period	06/27/18 → 06/29/18

Scopus Subject Areas

Artificial Intelligence
Computer Networks and Communications
Hardware and Architecture
Software
Control and Optimization
Information Systems and Management

Keywords

Big Data
Configuration
Hadoop
performance tuning

Access to Document

10.1109/SNPD.2018.8441059License: Unspecified

Cite this

Kim, J., Kancharla, A., Seol, J., Park, N. J., & Park, N. (2018). Optimized common parameter set extraction by benchmarking applications on a big data platform. In H. J. Hwang, L. Cai, G. H. Yeom, T. Matsuo, H. K. Kim, H. Yeo, C. S. Hong, N. Fukuta, T. Ito, & H. Miao (Eds.), Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018 Article 8441059 (Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SNPD.2018.8441059

Kim, Jongyeop ; Kancharla, Abhilash ; Seol, Jongho et al. / Optimized common parameter set extraction by benchmarking applications on a big data platform. Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018. editor / Ha Jin Hwang ; Lizhi Cai ; Gun Huck Yeom ; Tokuro Matsuo ; Haeng Kon Kim ; Hyun Yeo ; Chung Sun Hong ; Naoki Fukuta ; Takayuki Ito ; Huaikou Miao. Institute of Electrical and Electronics Engineers Inc., 2018. (Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018).

@inproceedings{ed5c5c8119134b938e762283258663ba,

title = "Optimized common parameter set extraction by benchmarking applications on a big data platform",

abstract = " This research proposes the methodology to extract common configuration parameter set by applying multiple benchmark applications including TeraSort., TestDFSIO, and MrBench on the Hadoop Distributed File System. In the process of determining parameter set for each stage, one parameter and its associated values selected which is reduced system performance in terms of overall execution time difference are measured by multiple applications on a Hadoop cluster. The experimental results demonstrate the proposed extended greedy manner provide a feasible benchmark model for the multiple tasks. In this way, we have found several parameter value sets that can reduce the execution time by 27% of the values provided by Hadoop default. ",

keywords = "Big Data, Configuration, Hadoop, performance tuning",

author = "Jongyeop Kim and Abhilash Kancharla and Jongho Seol and Park, {Noh Jin} and Nohpill Park",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018 ; Conference date: 27-06-2018 Through 29-06-2018",

year = "2018",

month = aug,

day = "20",

doi = "10.1109/SNPD.2018.8441059",

language = "English",

isbn = "9781538658895",

series = "Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

editor = "Hwang, {Ha Jin} and Lizhi Cai and Yeom, {Gun Huck} and Tokuro Matsuo and Kim, {Haeng Kon} and Hyun Yeo and Hong, {Chung Sun} and Naoki Fukuta and Takayuki Ito and Huaikou Miao",

booktitle = "Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018",

address = "United States",

}

Kim, J, Kancharla, A, Seol, J, Park, NJ & Park, N 2018, Optimized common parameter set extraction by benchmarking applications on a big data platform. in HJ Hwang, L Cai, GH Yeom, T Matsuo, HK Kim, H Yeo, CS Hong, N Fukuta, T Ito & H Miao (eds), Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018., 8441059, Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018, Institute of Electrical and Electronics Engineers Inc., 19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018, Busan, Korea, Republic of, 06/27/18. https://doi.org/10.1109/SNPD.2018.8441059

Optimized common parameter set extraction by benchmarking applications on a big data platform. / Kim, Jongyeop; Kancharla, Abhilash; Seol, Jongho et al.
Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018. ed. / Ha Jin Hwang; Lizhi Cai; Gun Huck Yeom; Tokuro Matsuo; Haeng Kon Kim; Hyun Yeo; Chung Sun Hong; Naoki Fukuta; Takayuki Ito; Huaikou Miao. Institute of Electrical and Electronics Engineers Inc., 2018. 8441059 (Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018).

Research output: Contribution to book or proceeding › Conference article › peer-review

TY - GEN

T1 - Optimized common parameter set extraction by benchmarking applications on a big data platform

AU - Kim, Jongyeop

AU - Kancharla, Abhilash

AU - Seol, Jongho

AU - Park, Noh Jin

AU - Park, Nohpill

PY - 2018/8/20

Y1 - 2018/8/20

N2 - This research proposes the methodology to extract common configuration parameter set by applying multiple benchmark applications including TeraSort., TestDFSIO, and MrBench on the Hadoop Distributed File System. In the process of determining parameter set for each stage, one parameter and its associated values selected which is reduced system performance in terms of overall execution time difference are measured by multiple applications on a Hadoop cluster. The experimental results demonstrate the proposed extended greedy manner provide a feasible benchmark model for the multiple tasks. In this way, we have found several parameter value sets that can reduce the execution time by 27% of the values provided by Hadoop default.

AB - This research proposes the methodology to extract common configuration parameter set by applying multiple benchmark applications including TeraSort., TestDFSIO, and MrBench on the Hadoop Distributed File System. In the process of determining parameter set for each stage, one parameter and its associated values selected which is reduced system performance in terms of overall execution time difference are measured by multiple applications on a Hadoop cluster. The experimental results demonstrate the proposed extended greedy manner provide a feasible benchmark model for the multiple tasks. In this way, we have found several parameter value sets that can reduce the execution time by 27% of the values provided by Hadoop default.

KW - Big Data

KW - Configuration

KW - Hadoop

KW - performance tuning

UR - http://www.scopus.com/inward/record.url?scp=85053522536&partnerID=8YFLogxK

U2 - 10.1109/SNPD.2018.8441059

DO - 10.1109/SNPD.2018.8441059

M3 - Conference article

SN - 9781538658895

T3 - Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018

BT - Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018

A2 - Hwang, Ha Jin

A2 - Cai, Lizhi

A2 - Yeom, Gun Huck

A2 - Matsuo, Tokuro

A2 - Kim, Haeng Kon

A2 - Yeo, Hyun

A2 - Hong, Chung Sun

A2 - Fukuta, Naoki

A2 - Ito, Takayuki

A2 - Miao, Huaikou

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018

Y2 - 27 June 2018 through 29 June 2018

ER -

Kim J, Kancharla A, Seol J, Park NJ, Park N. Optimized common parameter set extraction by benchmarking applications on a big data platform. In Hwang HJ, Cai L, Yeom GH, Matsuo T, Kim HK, Yeo H, Hong CS, Fukuta N, Ito T, Miao H, editors, Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018. Institute of Electrical and Electronics Engineers Inc. 2018. 8441059. (Proceedings - 2018 IEEE/ACIS 19th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD 2018). doi: 10.1109/SNPD.2018.8441059

Optimized common parameter set extraction by benchmarking applications on a big data platform

Abstract

Publication series

Conference

Scopus Subject Areas

Keywords

Access to Document

Other files and links

Fingerprint

Cite this