Robust understanding in multimodal interfaces

  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Declaration of views_handler_filter::options_validate() should be compatible with views_handler::options_validate($form, &$form_state) in /home/speechmap/stcspeechmap.org/Drupal/modules/views/handlers/views_handler_filter.inc on line 589.
  • strict warning: Declaration of views_handler_filter::options_submit() should be compatible with views_handler::options_submit($form, &$form_state) in /home/speechmap/stcspeechmap.org/Drupal/modules/views/handlers/views_handler_filter.inc on line 589.
  • strict warning: Declaration of views_plugin_style_default::options() should be compatible with views_object::options() in /home/speechmap/stcspeechmap.org/Drupal/modules/views/plugins/views_plugin_style_default.inc on line 25.
  • strict warning: Declaration of views_plugin_row::options_validate() should be compatible with views_plugin::options_validate(&$form, &$form_state) in /home/speechmap/stcspeechmap.org/Drupal/modules/views/plugins/views_plugin_row.inc on line 135.
  • strict warning: Declaration of views_plugin_row::options_submit() should be compatible with views_plugin::options_submit(&$form, &$form_state) in /home/speechmap/stcspeechmap.org/Drupal/modules/views/plugins/views_plugin_row.inc on line 135.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Declaration of views_handler_argument::init() should be compatible with views_handler::init(&$view, $options) in /home/speechmap/stcspeechmap.org/Drupal/modules/views/handlers/views_handler_argument.inc on line 745.
  • strict warning: Declaration of views_handler_filter_node_status::operator_form() should be compatible with views_handler_filter::operator_form(&$form, &$form_state) in /home/speechmap/stcspeechmap.org/Drupal/modules/views/modules/node/views_handler_filter_node_status.inc on line 14.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
  • strict warning: Non-static method view::load() should not be called statically in /home/speechmap/stcspeechmap.org/Drupal/modules/views/views.module on line 879.
TitleRobust understanding in multimodal interfaces
Publication TypeJournal Article
Year of Publication2009
AuthorsBangalore, S, Johnston M
JournalComputational Linguistics
Volume35
Pagination345–397
Abstract

Multimodal grammars provide an effective mechanism for quickly creating integration and understanding capabilities for interactive systems supporting simultaneous use of multiple input modalities. However, like other approaches based on hand-crafted grammars, multimodal grammars can be brittle with respect to unexpected, erroneous, or disfluent input. In this article, we show how the finite-state approach to multimodal language processing can be extended to support multimodal applications combining speech with complex freehand pen input, and evaluate the approach in the context of a multimodal conversational system (MATCH). We explore a range of different techniques for improving the robustness of multimodal integration and understanding. These include techniques for building effective language models for speech recognition when little or no multimodal training data is available, and techniques for robust multimodal understanding that draw on classification, machine translation, and sequence edit methods. We also explore the use of edit-based methods to overcome mismatches between the gesture stream and the speech stream.

URLhttp://uceresource.org/articles/CS/2009/ACL-09-Vol-35-3-2.pdf