Automated Configuration Parameter Classfication Model for Hive Query Plan on the Apache Yarn

Jongyeop Kim, Seongsoo Kim, Donghoon Kim, Hong Liu

Research output: Contribution to book or proceedingConference articlepeer-review

Abstract

This research proposed an automated configuration parameter classification model to arrange optimized Hive Query processing environment on the Apache Hadoop Distributed File System. In this model, the Analysis statistic command issued to measuring expected performance for the Hive tables on the Hadoop yarn platform with varying combinations of parameter configuration. The e-heuristic methodology is applied to effectively shrinking parameter search space during automated tuning process. We controlled the transition between evaluation spaces using one main parameter and one auxiliary parameter that are expected to reach the global optimum in each evaluation space. This model identifies the Hive parameters that access Hive table optimally and expects to improve query execution time by 15% against to the default Hive settings.
Original languageEnglish
Title of host publicationProceedings - 2019 IEEE/ACIS 4th International Conference on Big Data, Cloud Computing, and Data Science, BCD 2019
EditorsMotoi Iwashita, Atsushi Shimoda, Prajak Chertchom
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages77-83
Number of pages7
ISBN (Electronic)9781728108865
DOIs
StatePublished - May 2019
Event4th IEEE/ACIS International Conference on Big Data, Cloud Computing, and Data Science, BCD 2019 - Honolulu, United States
Duration: May 29 2019May 31 2019

Publication series

NameProceedings - 2019 IEEE/ACIS 4th International Conference on Big Data, Cloud Computing, and Data Science, BCD 2019

Conference

Conference4th IEEE/ACIS International Conference on Big Data, Cloud Computing, and Data Science, BCD 2019
Country/TerritoryUnited States
CityHonolulu
Period05/29/1905/31/19

Scopus Subject Areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Information Systems and Management

Keywords

  • Hive configuration
  • Hive parameter
  • Hive tuning

Fingerprint

Dive into the research topics of 'Automated Configuration Parameter Classfication Model for Hive Query Plan on the Apache Yarn'. Together they form a unique fingerprint.

Cite this