2021 IEEE International Conference on Acoustics, Speech and Signal Processing
6-11 June 2021 • Toronto, Ontario, Canada
Extracting Knowledge from Information
| Paper ID | MLSP-20.1 | ||
| Paper Title | CONTINUOUS-TIME SELF-ATTENTION IN NEURAL DIFFERENTIAL EQUATION | ||
| Authors | Jen-Tzung Chien, Yi-Hsiang Chen, National Chiao Tung University, Taiwan | ||
| Session | MLSP-20: Attention and Autoencoder Networks | ||
| Location | Gather.Town | ||
| Session Time: | Wednesday, 09 June, 15:30 - 16:15 | ||
| Presentation Time: | Wednesday, 09 June, 15:30 - 16:15 | ||
| Presentation | Poster | ||
| Topic | Machine Learning for Signal Processing: [MLR-SLER] Sequential learning; sequential decision methods | ||
| IEEE Xplore Open Preview | Click here to view in IEEE Xplore | ||
| Abstract | Neural differential equation (NDE) is recently developed as a continuous-time state machine which can faithfully represent the irregularly-sampled sequence data. NDE is seen as a substantial extension of recurrent neural network (RNN) which conducts discrete-time state representation for regularly-sampled data. This study presents a new continuous-time attention to improve sequential learning where the region of interest in continuous-time state trajectory over observed as well as missing samples is sufficiently attended. However, the attention score, calculated by relating between a query and a sequence, is memory demanding because self-attention should treat all time observations as query vectors to feed them into ordinary differential equation (ODE) solver. To deal with this issue, we develop a new form of dynamics for continuous-time attention where the causality property is adopted such that query vector is fed into ODE solver up to current time. The experiments on irregularly-sampled human activities and medical features show that this method obtains desirable performance with efficient memory consumption. | ||