Parallel mining of Top-k high utility itemsets in spark in-memory computing architecture

Chun Han Lin, Cheng Wei Wu, Jian Tao Huang, Vincent Shin-Mu Tseng*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Top-k high utility itemset (abbr. Top-k HUI) mining aims at efficiently mining k itemsets having the highest utility without setting the minimum utility thresholds. Although some studies have been conducted on top-k HUI mining recently, they mainly focus on centralized databases and are not scalable for big data environments. To address the above issues, this paper proposes a novel framework for parallel mining of top-k high utility itemsets in big data. Besides, a new algorithm called PKU (Parallel Top-K High Utility Itemset Mining) is proposed for parallel mining of top-k HUIs on Spark in-memory platform. It adopts MapReduce architecture to divide the whole mining task into several independent subtasks, and takes good use of Spark in-memory computing technology for efficiently processing data in parallel. Moreover, several novel strategies are also proposed for pruning the redundant candidates such that the execution time and memory usage in the mining process are reduced greatly. The proposed PKU algorithm inherits several advantages of Spark, including low communication cost, fault tolerance, and high scalability. Experimental results on both real and synthetic datasets show that PKU has good scalability and performance on large datasets with outperforming several benchmarking algorithms.

Original languageEnglish
Title of host publicationAdvances in Knowledge Discovery and Data Mining - 23rd Pacific-Asia Conference, PAKDD 2019, Proceedings
EditorsZhi-Hua Zhou, Min-Ling Zhang, Sheng-Jun Huang, Qiang Yang, Zhiguo Gong
PublisherSpringer Verlag
Pages253-265
Number of pages13
ISBN (Print)9783030161446
DOIs
StatePublished - 1 Jan 2019
Event23rd Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2019 - Macau, China
Duration: 14 Apr 201917 Apr 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11440 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference23rd Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2019
CountryChina
CityMacau
Period14/04/1917/04/19

Keywords

  • Big data
  • In-memory computing
  • MapReduce
  • Spark platform
  • Top-k high utility itemset

Fingerprint Dive into the research topics of 'Parallel mining of Top-k high utility itemsets in spark in-memory computing architecture'. Together they form a unique fingerprint.

  • Cite this

    Lin, C. H., Wu, C. W., Huang, J. T., & Tseng, V. S-M. (2019). Parallel mining of Top-k high utility itemsets in spark in-memory computing architecture. In Z-H. Zhou, M-L. Zhang, S-J. Huang, Q. Yang, & Z. Gong (Eds.), Advances in Knowledge Discovery and Data Mining - 23rd Pacific-Asia Conference, PAKDD 2019, Proceedings (pp. 253-265). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11440 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-030-16145-3_20