Tensor-factorized neural networks

Jen-Tzung Chien*, Yi Ting Bao

*Corresponding author for this work

Research output: Contribution to journalArticle

17 Scopus citations

Abstract

The growing interests in multiway data analysis and deep learning have drawn tensor factorization (TF) and neural network (NN) as the crucial topics. Conventionally, the NN model is estimated from a set of one-way observations. Such a vectorized NN is not generalized for learning the representation from multiway observations. The classification performance using vectorized NN is constrained, because the temporal or spatial information in neighboring ways is disregarded. More parameters are required to learn the complicated data structure. This paper presents a new tensor-factorized NN (TFNN), which tightly integrates TF and NN for multiway feature extraction and classification under a unified discriminative objective. This TFNN is seen as a generalized NN, where the affine transformation in an NN is replaced by the multilinear and multiway factorization for tensor-based NN. The multiway information is preserved through layerwise factorization. Tucker decomposition and nonlinear activation are performed in each hidden layer. The tensor-factorized error backpropagation is developed to train TFNN with the limited parameter size and computation time. This TFNN can be further extended to realize the convolutional TFNN (CTFNN) by looking at small subtensors through the factorized convolution. Experiments on real-world classification tasks demonstrate that TFNN and CTFNN attain substantial improvement when compared with an NN and a convolutional NN, respectively.

Original languageEnglish
Pages (from-to)1998-2011
Number of pages14
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume29
Issue number5
DOIs
StatePublished - 1 May 2018

Keywords

  • And tensor-factorized error backpropagation
  • Neural network (NN)
  • Pattern classification
  • Tensor factorization (TF)

Fingerprint Dive into the research topics of 'Tensor-factorized neural networks'. Together they form a unique fingerprint.

Cite this