A fast and low idle time method for mining frequent patterns in distributed and many-task computing environments

Chun-Cheng Lin, Sheng Hao Chung, Ju Chin Chen, Yuan Tse Yu, Kawuu W. Lin*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Association rules mining has attracted much attention among data mining topics because it has been successfully applied in various fields to find the association between purchased items by identifying frequent patterns (FPs). Currently, databases are huge, ranging in size from terabytes to petabytes. Although past studies can effectively discover FPs to deduce association rules, the execution efficiency is still a critical problem, particularly for big data. Progressive size working set (PSWS) and parallel FP-growth (PFP) are state-of-the-art methods that have been applied successfully to parallel and distributed computing technology to improve mining processing time in many-task computing, thereby bridging the gap between high-throughput and high-performance computing. However, such methods cannot mine before obtaining a complete FP-tree or the corresponding subdatabase, causing a high idle time for computing nodes. We propose a method that can begin mining when a small part of an FP-tree is received. The idle time of computing nodes can be reduced, and thus, the time required for mining can be reduced effectively. Through an empirical evaluation, the proposed method is shown to be faster than PSWS and PFP.

Original languageEnglish
Pages (from-to)613-641
Number of pages29
JournalDistributed and Parallel Databases
Volume36
Issue number4
DOIs
StatePublished - 1 Dec 2018

Keywords

  • Distributed computing
  • Distributed mining
  • Frequent pattern mining
  • Many-task computing

Fingerprint Dive into the research topics of 'A fast and low idle time method for mining frequent patterns in distributed and many-task computing environments'. Together they form a unique fingerprint.

Cite this