| Presentation # | 3 |
| Session: | Spoken Language Understanding |
| Location: | Kallirhoe Hall |
| Session Time: | Wednesday, December 19, 10:00 - 12:00 |
| Presentation Time: | Wednesday, December 19, 10:00 - 12:00 |
| Presentation: |
Poster
|
| Topic: |
Spoken language understanding: |
| Paper Title: |
A Re-ranker Scheme for Integrating Large Scale NLU models |
| Authors: |
Chengwei Su, Rahul Gupta, Shankar Ananthakrishnan, Spyros Matsoukas, Amazon, United States |
| Abstract: |
Large scale Natural Language Understanding (NLU) systems are typically trained on large quantities of data, requiring a fast and scalable training strategy. A typical design for NLU systems consists of domain-level NLU modules (domain classifier, intent classifier and named entity recognizer). Hypotheses (NLU interpretations consisting of various intent+slot combinations) from these domain specific modules are typically aggregated with another downstream component. The re-ranker integrates outputs from domain-level recognizers, returning a scored list of cross domain hypotheses. An ideal re-ranker will exhibit the following two properties: (a) it should prefer the most relevant hypothesis for the given input as the top hypothesis and, (b) the interpretation scores corresponding to each hypothesis produced by the re-ranker should be calibrated. Calibration allows the final NLU interpretation score to be comparable across domains. We propose a novel re-ranker strategy that addresses these aspects, while also maintaining domain specific modularity. We design optimization loss functions for such a modularized re-ranker and present results on decreasing the top hypothesis error rate as well as maintaining the model calibration. We also experiment with an extension involving training the domain specific re-rankers on datasets curated independently by each domain to allow further asynchronization. |