Parallel mining of Top-k high utility itemsets in spark in-memory computing architecture

Chun Han Lin, Cheng Wei Wu, Jian Tao Huang, Vincent Shin-Mu Tseng*

*Corresponding author for this work

研究成果: Conference contribution

摘要

Top-k high utility itemset (abbr. Top-k HUI) mining aims at efficiently mining k itemsets having the highest utility without setting the minimum utility thresholds. Although some studies have been conducted on top-k HUI mining recently, they mainly focus on centralized databases and are not scalable for big data environments. To address the above issues, this paper proposes a novel framework for parallel mining of top-k high utility itemsets in big data. Besides, a new algorithm called PKU (Parallel Top-K High Utility Itemset Mining) is proposed for parallel mining of top-k HUIs on Spark in-memory platform. It adopts MapReduce architecture to divide the whole mining task into several independent subtasks, and takes good use of Spark in-memory computing technology for efficiently processing data in parallel. Moreover, several novel strategies are also proposed for pruning the redundant candidates such that the execution time and memory usage in the mining process are reduced greatly. The proposed PKU algorithm inherits several advantages of Spark, including low communication cost, fault tolerance, and high scalability. Experimental results on both real and synthetic datasets show that PKU has good scalability and performance on large datasets with outperforming several benchmarking algorithms.

原文English
主出版物標題Advances in Knowledge Discovery and Data Mining - 23rd Pacific-Asia Conference, PAKDD 2019, Proceedings
編輯Zhi-Hua Zhou, Min-Ling Zhang, Sheng-Jun Huang, Qiang Yang, Zhiguo Gong
發行者Springer Verlag
頁面253-265
頁數13
ISBN(列印)9783030161446
DOIs
出版狀態Published - 1 一月 2019
事件23rd Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2019 - Macau, China
持續時間: 14 四月 201917 四月 2019

出版系列

名字Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
11440 LNAI
ISSN(列印)0302-9743
ISSN(電子)1611-3349

Conference

Conference23rd Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2019
國家China
城市Macau
期間14/04/1917/04/19

指紋 深入研究「Parallel mining of Top-k high utility itemsets in spark in-memory computing architecture」主題。共同形成了獨特的指紋。

  • 引用此

    Lin, C. H., Wu, C. W., Huang, J. T., & Tseng, V. S-M. (2019). Parallel mining of Top-k high utility itemsets in spark in-memory computing architecture. 於 Z-H. Zhou, M-L. Zhang, S-J. Huang, Q. Yang, & Z. Gong (編輯), Advances in Knowledge Discovery and Data Mining - 23rd Pacific-Asia Conference, PAKDD 2019, Proceedings (頁 253-265). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); 卷 11440 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-030-16145-3_20