| Paper ID | HLT-16.2 |
| Paper Title |
MAKING PUNCTUATION RESTORATION ROBUST AND FAST WITH MULTI-TASK LEARNING AND KNOWLEDGE DISTILLATION |
| Authors |
Michael Hentschel, Emiru Tsunoo, Takao Okuda, Sony Corporation, Japan |
| Session | HLT-16: Applications in Natural Language |
| Location | Gather.Town |
| Session Time: | Thursday, 10 June, 16:30 - 17:15 |
| Presentation Time: | Thursday, 10 June, 16:30 - 17:15 |
| Presentation |
Poster
|
| Topic |
Human Language Technology: [HLT-MLMD] Machine Learning Methods for Language |
| IEEE Xplore Open Preview |
Click here to view in IEEE Xplore |
| Virtual Presentation |
Click here to watch in the Virtual Conference |
| Abstract |
In punctuation restoration, we try to recover the missing punctuation from automatic speech recognition output to improve understandability. Currently, large pre-trained transformers such as BERT set the benchmark on this task but there are two main drawbacks to these models. First, the pre-training data does not match the output data from speech recognition that contains errors. Second, the large number of model parameters increases inference time. To address the former, we use a multi-task learning framework with ELECTRA, a recently proposed improvement on BERT, that has a generator-discriminator structure. The generator allows us to inject errors into the training data and, as our experiments show, this improves robustness against speech recognition errors during inference. To address the latter, we investigate knowledge distillation and parameter pruning of ELECTRA. In our experiments on the IWSLT 2012 benchmark data, a model with less than 11% the size of BERT achieved better performance while having an 82% faster inference time. |
