Silvia Miksch talking about time oriented visual analytics

It seems this term we picked a good slot for the lecture. On Thursday we had Prof. Silvia Miksch from Vienna University of Technology visiting our institute. We took this chance for another guest lecture in my advanced HCI class. Silvia presented a talk with the title “A Matter of Time: Interactive Visual Analytics of Time-Oriented Data and Information”. She first introduced the notion of interactive visual analytics and then systematically showed how time oriented data can be visually presented.

I really liked how Silvia motivated visual analytics and could not resist to adapt it with a Christmas theme. The picture shows three representations (1) numbers, always 3 grouped together, (2) a plot of the numbers where the first is the label and the second and the third are coordinates, and (3) a line connecting the labels in order. Her example was much nicer, but I missed to take a photo. And it is obvious that you do not put it on the same slide… Nevertheless I think even this simple Christmas tree example shows the power of visual analytics. This will go in my slide set for presentations in schools ;-)

If you are more interested in the details of the visualization of time oriented data, please have a look at the following book: Visallization of Time-Oriented Data, by Wolfgang Aigner, Silvia Miksch, Heidrun Schumann, and Christian Tominski. Springer, 2011. http://www.timeviz.net [2]. After the talk there was an interested discussion about the relationship and fundamental difference between time and space. I think this is worthwhile further discussion.

Another direction to follow up is tangible (visual) analytics. It would be interesting to assess the contributions to understanding of further modalities when interactively exploring data, e.g. haptics and sound. Some years back Martin Schrittenloher (one of my students in Munich) visited Morten Fjeld for his project thesis and experimented with force feedback sliders [1], … perhaps we should have this as a project topic again! An approach would be to look specifically at the understanding of data when force-feedback is presented on certain dimensions.

References
[1] Jenaro, J., Shahrokni, A., Schrittenloher, and M., Fjeld, M. 2007. One-Dimensional Force Feedback Slider: Digital platform. In Proc. Workshop at the IEEE Virtual Reality 2007 Conference: Mixed Reality User Interfaces: Specification, Authoring, Adaptation (MRUI07), 47-51
[2] Wolfgang Aigner, Silvia Miksch, Heidrun Schumann, and Christian Tominski. Visallization of Time-Oriented Data. Springer, 2011. http://www.timeviz.net

Silvia Miksch talking about time oriented visual analytics

It seems this term we picked a good slot for the lecture. On Thursday we had Prof. Silvia Miksch from Vienna University of Technology visiting our institute. We took this chance for another guest lecture in my advanced HCI class. Silvia presented a talk with the title “A Matter of Time: Interactive Visual Analytics of Time-Oriented Data and Information”. She first introduced the notion of interactive visual analytics and then systematically showed how time oriented data can be visually presented.

I really liked how Silvia motivated visual analytics and could not resist to adapt it with a Christmas theme. The picture shows three representations (1) numbers, always 3 grouped together, (2) a plot of the numbers where the first is the label and the second and the third are coordinates, and (3) a line connecting the labels in order. Her example was much nicer, but I missed to take a photo. And it is obvious that you do not put it on the same slide… Nevertheless I think even this simple Christmas tree example shows the power of visual analytics. This will go in my slide set for presentations in schools ;-)

If you are more interested in the details of the visualization of time oriented data, please have a look at the following book: Visallization of Time-Oriented Data, by Wolfgang Aigner, Silvia Miksch, Heidrun Schumann, and Christian Tominski. Springer, 2011. http://www.timeviz.net [2]. After the talk there was an interested discussion about the relationship and fundamental difference between time and space. I think this is worthwhile further discussion.

Another direction to follow up is tangible (visual) analytics. It would be interesting to assess the contributions to understanding of further modalities when interactively exploring data, e.g. haptics and sound. Some years back Martin Schrittenloher (one of my students in Munich) visited Morten Fjeld for his project thesis and experimented with force feedback sliders [1], … perhaps we should have this as a project topic again! An approach would be to look specifically at the understanding of data when force-feedback is presented on certain dimensions.

References
[1] Jenaro, J., Shahrokni, A., Schrittenloher, and M., Fjeld, M. 2007. One-Dimensional Force Feedback Slider: Digital platform. In Proc. Workshop at the IEEE Virtual Reality 2007 Conference: Mixed Reality User Interfaces: Specification, Authoring, Adaptation (MRUI07), 47-51
[2] Wolfgang Aigner, Silvia Miksch, Heidrun Schumann, and Christian Tominski. Visallization of Time-Oriented Data. Springer, 2011. http://www.timeviz.net

Bryan Reimer: Opening keynote at Auto-UI 2011 in Salzburg

Bryan started his keynote talk the automotive user interface conference (auto-ui.org) in Salzburg with reminding us that having controversial discussions about the HMI in the car is not new. Quoting a newspaper article from the 1930s on the introduction of the radio in the car and its impact on the driver he picked an interesting example, that can be seen as the root of many issues we have now with infotainment systems in the car.

The central question he raised is: how to create user interface that fit human users? He made an important point: humans are not “designed” to drive at high speed in complex environments; perception has evolved for walking and running in natural environment. Additionally to the basic limitations of human cognition, there is a great variety of capabilities of drivers, their skills and cognitive ability (e.g. influence of age). A implication of the global change is demographics is that the average capabilities of a drivers will be reduced – basically as many older people will be drivers…

Over the last 100 years cars have changes significantly! Looking more closely Bryan argues that much of the chance happened in the last 10 years. There has been little change from the 1950s to the 1990s with regard to the car user interface.

It is apparent that secondary tasks are becoming more important to the user. Users will interact more while driving because the can. It is however not obvious that they are capable of it.

Even given these developments it is apparent that driving has become safer. Passive safety has been improved massively and this made driving much safer. There seems to be a drawback to this as well, as people may take greater risks as they feel safer. The next step is really to avoid accidence in the first place. Bryan argues that the interaction between driver, environment, and vehicles is very important in that. He suggests that we should make more of an effort to create systems that fit the drivers.

The Yerkes-Dodson Law helps to understand how to design systems that keep peoples attention in the optimal performance. He made an important point: there are certain issues that cannot be solved, e.g. if someone is tired we can do only very little – the driver will need to rest. We should make sure that we take these things into account when designing systems.

Visual distraction is an obvious factor and much discussed in the papers at the conference – but Bryan argued that “eyes on the road” is not equal to “mind on the road”. I think this is really a very important point. Ensuring that people keep their eyes on the road, seeing things is not enough. The big resulting question is how to keep or get people focused on the street and environment. It seems there is some more research to do…

The variety of interfaces and interaction metaphors build into cars opens more choices but at the same time creates problems, as people need to learn and understand them. A simple question such as: How do you switch the car off? may be hard to answer (Bryan had the example of a car with a push button starter, where you cannot remove the key). I think there are simple questions that can be learned from industry and production machines… add an emergency stop button and make it mandatory ;-)

If you are interested more about Bryan’s work look at his webpage or his page at the MIT agelab or one of his recent publications [1] in the IEEE Pervasive Computing Magazine’s special issue on automotive computing, see [2] for an introduction to the special issue.

Sorry for the poor quality photos … back row and an iPhone…

[1] Joseph F. Coughlin, Bryan Reimer, and Bruce Mehler. 2011. Monitoring, Managing, and Motivating Driver Safety and Well-Being. IEEE Pervasive Computing 10, 3 (July 2011), 14-21. DOI=10.1109/MPRV.2011.54 http://dx.doi.org/10.1109/MPRV.2011.54

[2] Albrecht Schmidt, Joseph Paradiso, and Brian Noble. 2011. Automotive Pervasive Computing. IEEE Pervasive Computing 10, 3 (July 2011), 12-13. DOI=10.1109/MPRV.2011.45 http://dx.doi.org/10.1109/MPRV.2011.45

>Bryan Reimer: Opening keynote at Auto-UI 2011 in Salzburg

>Bryan started his keynote talk the automotive user interface conference (auto-ui.org) in Salzburg with reminding us that having controversial discussions about the HMI in the car is not new. Quoting a newspaper article from the 1930s on the introduction of the radio in the car and its impact on the driver he picked an interesting example, that can be seen as the root of many issues we have now with infotainment systems in the car.

The central question he raised is: how to create user interface that fit human users? He made an important point: humans are not “designed” to drive at high speed in complex environments; perception has evolved for walking and running in natural environment. Additionally to the basic limitations of human cognition, there is a great variety of capabilities of drivers, their skills and cognitive ability (e.g. influence of age). A implication of the global change is demographics is that the average capabilities of a drivers will be reduced – basically as many older people will be drivers…

Over the last 100 years cars have changes significantly! Looking more closely Bryan argues that much of the chance happened in the last 10 years. There has been little change from the 1950s to the 1990s with regard to the car user interface.

It is apparent that secondary tasks are becoming more important to the user. Users will interact more while driving because the can. It is however not obvious that they are capable of it.

Even given these developments it is apparent that driving has become safer. Passive safety has been improved massively and this made driving much safer. There seems to be a drawback to this as well, as people may take greater risks as they feel safer. The next step is really to avoid accidence in the first place. Bryan argues that the interaction between driver, environment, and vehicles is very important in that. He suggests that we should make more of an effort to create systems that fit the drivers.

The Yerkes-Dodson Law helps to understand how to design systems that keep peoples attention in the optimal performance. He made an important point: there are certain issues that cannot be solved, e.g. if someone is tired we can do only very little – the driver will need to rest. We should make sure that we take these things into account when designing systems.

Visual distraction is an obvious factor and much discussed in the papers at the conference – but Bryan argued that “eyes on the road” is not equal to “mind on the road”. I think this is really a very important point. Ensuring that people keep their eyes on the road, seeing things is not enough. The big resulting question is how to keep or get people focused on the street and environment. It seems there is some more research to do…

The variety of interfaces and interaction metaphors build into cars opens more choices but at the same time creates problems, as people need to learn and understand them. A simple question such as: How do you switch the car off? may be hard to answer (Bryan had the example of a car with a push button starter, where you cannot remove the key). I think there are simple questions that can be learned from industry and production machines… add an emergency stop button and make it mandatory ;-)

If you are interested more about Bryan’s work look at his webpage or his page at the MIT agelab or one of his recent publications [1] in the IEEE Pervasive Computing Magazine’s special issue on automotive computing, see [2] for an introduction to the special issue.

Sorry for the poor quality photos … back row and an iPhone…

[1] Joseph F. Coughlin, Bryan Reimer, and Bruce Mehler. 2011. Monitoring, Managing, and Motivating Driver Safety and Well-Being. IEEE Pervasive Computing 10, 3 (July 2011), 14-21. DOI=10.1109/MPRV.2011.54 http://dx.doi.org/10.1109/MPRV.2011.54

[2] Albrecht Schmidt, Joseph Paradiso, and Brian Noble. 2011. Automotive Pervasive Computing. IEEE Pervasive Computing 10, 3 (July 2011), 12-13. DOI=10.1109/MPRV.2011.45 http://dx.doi.org/10.1109/MPRV.2011.45

Bryan Reimer: Opening keynote at Auto-UI 2011 in Salzburg

Bryan started his keynote talk the automotive user interface conference (auto-ui.org) in Salzburg with reminding us that having controversial discussions about the HMI in the car is not new. Quoting a newspaper article from the 1930s on the introduction of the radio in the car and its impact on the driver he picked an interesting example, that can be seen as the root of many issues we have now with infotainment systems in the car.

The central question he raised is: how to create user interface that fit human users? He made an important point: humans are not “designed” to drive at high speed in complex environments; perception has evolved for walking and running in natural environment. Additionally to the basic limitations of human cognition, there is a great variety of capabilities of drivers, their skills and cognitive ability (e.g. influence of age). A implication of the global change is demographics is that the average capabilities of a drivers will be reduced – basically as many older people will be drivers…

Over the last 100 years cars have changes significantly! Looking more closely Bryan argues that much of the chance happened in the last 10 years. There has been little change from the 1950s to the 1990s with regard to the car user interface.

It is apparent that secondary tasks are becoming more important to the user. Users will interact more while driving because the can. It is however not obvious that they are capable of it.

Even given these developments it is apparent that driving has become safer. Passive safety has been improved massively and this made driving much safer. There seems to be a drawback to this as well, as people may take greater risks as they feel safer. The next step is really to avoid accidence in the first place. Bryan argues that the interaction between driver, environment, and vehicles is very important in that. He suggests that we should make more of an effort to create systems that fit the drivers.

The Yerkes-Dodson Law helps to understand how to design systems that keep peoples attention in the optimal performance. He made an important point: there are certain issues that cannot be solved, e.g. if someone is tired we can do only very little – the driver will need to rest. We should make sure that we take these things into account when designing systems.

Visual distraction is an obvious factor and much discussed in the papers at the conference – but Bryan argued that “eyes on the road” is not equal to “mind on the road”. I think this is really a very important point. Ensuring that people keep their eyes on the road, seeing things is not enough. The big resulting question is how to keep or get people focused on the street and environment. It seems there is some more research to do…

The variety of interfaces and interaction metaphors build into cars opens more choices but at the same time creates problems, as people need to learn and understand them. A simple question such as: How do you switch the car off? may be hard to answer (Bryan had the example of a car with a push button starter, where you cannot remove the key). I think there are simple questions that can be learned from industry and production machines… add an emergency stop button and make it mandatory ;-)

If you are interested more about Bryan’s work look at his webpage or his page at the MIT agelab or one of his recent publications [1] in the IEEE Pervasive Computing Magazine’s special issue on automotive computing, see [2] for an introduction to the special issue.

Sorry for the poor quality photos … back row and an iPhone…

[1] Joseph F. Coughlin, Bryan Reimer, and Bruce Mehler. 2011. Monitoring, Managing, and Motivating Driver Safety and Well-Being. IEEE Pervasive Computing 10, 3 (July 2011), 14-21. DOI=10.1109/MPRV.2011.54 http://dx.doi.org/10.1109/MPRV.2011.54

[2] Albrecht Schmidt, Joseph Paradiso, and Brian Noble. 2011. Automotive Pervasive Computing. IEEE Pervasive Computing 10, 3 (July 2011), 12-13. DOI=10.1109/MPRV.2011.45 http://dx.doi.org/10.1109/MPRV.2011.45

Opening Keynote at AMI 2011: Margaret Morris

Margaret (Margi) Morris presented the opening keynote at the 2011 conference on ambient intelligence in Amsterdam (AMI2011) with the title “Left to our own devices”.

Margaret brought up an interesting point on motivation: Showing people what they lose is a stronger motivator than the prospective of gain. She made the point in order to implement this the depicted loss has to be very specific. She showed a facebook applicationWith a little help from my friends”, where this basic concept is applied.   I had recently seen a bill board adverting campaign for safe driving on motorways in Germany using this approach (basically showing the risk of loss of family).

In the talk several examples of devices and applications were presented. To learn more about her work I recommend the following two papers: at tool to improve emotional self-awareness [1] and an investing in social networks and their utility to promote health [2].

Another point that made me think was the question of how we design interventions. One conceptual example was about an obesity campaign. The official UK campaign starts out with the statement that obesity is a problem for 9 million kids. Her alternative is to provide instead of the information a specific hint about an opportunity for action for an individual (e.g. telling the kid when it leaves school in the afternoon: now is probably a good time to play soccer with your friends, as 16 of them like to play soccer). An open research question that relates to this seems to me to investigate the impact of information about the norm, e.g. how will it affect my behavior if I know that 70% of my friends think driving too fast is OK vs. if I know that only 20% find it acceptable. I think this could be further explored in the context of social networks to create interesting persuasive technologies.


There has been an interesting discussion after the talk. Norbert Streitz questioned if it is a good idea to ask people to engage more with digital devices (e.g. self monitoring one’s mood). The question is hinting that the engagement with the digital device keeps us from interaction in the “real” world. I think this separation is disappearing fast – making a phone call, listening to MP3, chatting with friends on facebook is for many of us real, we live in a world that is augmented by technology and the boundaries are bluring…

[1] Morris ME, Kathawala Q, Leen TK, Gorenstein EE, Guilak F, Labhard M, Deleeuw W. Mobile Therapy: Case Study Evaluations of a Cell Phone Application for Emotional Self-Awareness. Journal of Medical Internet Research 2010;12(2):e10. URL: http://www.jmir.org/2010/2/e10/

[2] Margaret E. Morris. 2005. Social Networks as Health Feedback Displays. IEEE Internet Computing 9, 5 (September 2005), 29-37. DOI=10.1109/MIC.2005.109 http://dx.doi.org/10.1109/MIC.2005.109

>Opening Keynote at AMI 2011: Margaret Morris

>Margaret (Margi) Morris presented the opening keynote at the 2011 conference on ambient intelligence in Amsterdam (AMI2011) with the title “Left to our own devices”.

Margaret brought up an interesting point on motivation: Showing people what they lose is a stronger motivator than the prospective of gain. She made the point in order to implement this the depicted loss has to be very specific. She showed a facebook applicationWith a little help from my friends”, where this basic concept is applied.   I had recently seen a bill board adverting campaign for safe driving on motorways in Germany using this approach (basically showing the risk of loss of family).

In the talk several examples of devices and applications were presented. To learn more about her work I recommend the following two papers: at tool to improve emotional self-awareness [1] and an investing in social networks and their utility to promote health [2].

Another point that made me think was the question of how we design interventions. One conceptual example was about an obesity campaign. The official UK campaign starts out with the statement that obesity is a problem for 9 million kids. Her alternative is to provide instead of the information a specific hint about an opportunity for action for an individual (e.g. telling the kid when it leaves school in the afternoon: now is probably a good time to play soccer with your friends, as 16 of them like to play soccer). An open research question that relates to this seems to me to investigate the impact of information about the norm, e.g. how will it affect my behavior if I know that 70% of my friends think driving too fast is OK vs. if I know that only 20% find it acceptable. I think this could be further explored in the context of social networks to create interesting persuasive technologies.


There has been an interesting discussion after the talk. Norbert Streitz questioned if it is a good idea to ask people to engage more with digital devices (e.g. self monitoring one’s mood). The question is hinting that the engagement with the digital device keeps us from interaction in the “real” world. I think this separation is disappearing fast – making a phone call, listening to MP3, chatting with friends on facebook is for many of us real, we live in a world that is augmented by technology and the boundaries are bluring…

[1] Morris ME, Kathawala Q, Leen TK, Gorenstein EE, Guilak F, Labhard M, Deleeuw W. Mobile Therapy: Case Study Evaluations of a Cell Phone Application for Emotional Self-Awareness. Journal of Medical Internet Research 2010;12(2):e10. URL: http://www.jmir.org/2010/2/e10/

[2] Margaret E. Morris. 2005. Social Networks as Health Feedback Displays. IEEE Internet Computing 9, 5 (September 2005), 29-37. DOI=10.1109/MIC.2005.109 http://dx.doi.org/10.1109/MIC.2005.109

Opening Keynote at AMI 2011: Margaret Morris

Margaret (Margi) Morris presented the opening keynote at the 2011 conference on ambient intelligence in Amsterdam (AMI2011) with the title “Left to our own devices”.

Margaret brought up an interesting point on motivation: Showing people what they lose is a stronger motivator than the prospective of gain. She made the point in order to implement this the depicted loss has to be very specific. She showed a facebook applicationWith a little help from my friends”, where this basic concept is applied.   I had recently seen a bill board adverting campaign for safe driving on motorways in Germany using this approach (basically showing the risk of loss of family).

In the talk several examples of devices and applications were presented. To learn more about her work I recommend the following two papers: at tool to improve emotional self-awareness [1] and an investing in social networks and their utility to promote health [2].

Another point that made me think was the question of how we design interventions. One conceptual example was about an obesity campaign. The official UK campaign starts out with the statement that obesity is a problem for 9 million kids. Her alternative is to provide instead of the information a specific hint about an opportunity for action for an individual (e.g. telling the kid when it leaves school in the afternoon: now is probably a good time to play soccer with your friends, as 16 of them like to play soccer). An open research question that relates to this seems to me to investigate the impact of information about the norm, e.g. how will it affect my behavior if I know that 70% of my friends think driving too fast is OK vs. if I know that only 20% find it acceptable. I think this could be further explored in the context of social networks to create interesting persuasive technologies.


There has been an interesting discussion after the talk. Norbert Streitz questioned if it is a good idea to ask people to engage more with digital devices (e.g. self monitoring one’s mood). The question is hinting that the engagement with the digital device keeps us from interaction in the “real” world. I think this separation is disappearing fast – making a phone call, listening to MP3, chatting with friends on facebook is for many of us real, we live in a world that is augmented by technology and the boundaries are bluring…

[1] Morris ME, Kathawala Q, Leen TK, Gorenstein EE, Guilak F, Labhard M, Deleeuw W. Mobile Therapy: Case Study Evaluations of a Cell Phone Application for Emotional Self-Awareness. Journal of Medical Internet Research 2010;12(2):e10. URL: http://www.jmir.org/2010/2/e10/

[2] Margaret E. Morris. 2005. Social Networks as Health Feedback Displays. IEEE Internet Computing 9, 5 (September 2005), 29-37. DOI=10.1109/MIC.2005.109 http://dx.doi.org/10.1109/MIC.2005.109

>SiMPE 2010, Keynote: Trends and Challenges in Mobile Interaction

>I was invited to give a keynote talk at the 5th Workshop on Speech in Mobile and Pervasive Environments that was held as a part of ACM MobileHCI 2010 in Lisbon, Portugal. Over the last years we looked at speech as an additional modality in the automotive user interface domain; besides this my experience with speech interfaces is limited.

My talk, with the title “Trends and Challenges in Mobile Interaction” looked at different issues in mobile interaction. In some parts I reflected on modalities and opportunities for speech interaction.

When characterizing mobile interaction I pointed out the following points:

  • Interacting while on the move
  • Interaction is one of the user’s tasks (besides others – e.g. walking, standing in a crowd)
  • Environment in which the interaction takes place changes (e.g. on the train with varying light and noise conditions)
  • Interruptions happen frequently (e.g. boarding the bus, crossing the road)
  • Application usage is short (typically seconds to minutes)

My small set of key issues and recommendations for mobile UIs is:

  • Simple and Understandable
  • Perceptive and Context-Aware
  • Unobtrusive, Embedded and Integrated
  • Low Cognitive Load and Peripheral Usage
  • Users want to be in Control (especially on the move)

The presentations and discussion at the workshop were very interesting and I got a number of ideas for multimodal user interfaces – including speech.

One issue that made me think more was the question about natural language speech vs. specific speech commands. A colleague pointed me to Speech Graffiti [1] / Universal Speech Interface at CMU. I wonder if it would make sense to invent a Human Computer Interaction language (with a simple grammar and a vocabulary) that we could teach in a course over several weeks (e.g. similar effort than touch typing on a QUERTY keyboard) or as a foreign language at school to have a new effective means for interaction. Could this make us more effictive in interacting with information? Or should we try harder to get natural languge interaction working? Looking at the way (experienced) people use Google we can see that people adapt very successfully – probably faster than systems improve…

From some of the talks it seems that “Push to talk” seems to be a real issue for users and a reason for many user related errors in speech systems. Users do not push at the appropriate time, especially when there are other tasks to do, and hence utterances are cut off at the start and end. I would guess continuous recording of the speech and using the “push to talk” only as an indicator where to search in the audio stream may be a solution.

[1] Tomko, S. and Rosenfeld, R. 2004. Speech graffiti vs. natural language: assessing the user experience. In Proceedings of HLT-NAACL 2004: Short Papers (Boston, Massachusetts, May 02 – 07, 2004). Human Language Technology Conference. Association for Computational Linguistics, Morristown, NJ, 73-76. http://www.cs.cmu.edu/~usi/papers/HLT04.pdf

SiMPE 2010, Keynote: Trends and Challenges in Mobile Interaction

I was invited to give a keynote talk at the 5th Workshop on Speech in Mobile and Pervasive Environments that was held as a part of ACM MobileHCI 2010 in Lisbon, Portugal. Over the last years we looked at speech as an additional modality in the automotive user interface domain; besides this my experience with speech interfaces is limited.

My talk, with the title “Trends and Challenges in Mobile Interaction” looked at different issues in mobile interaction. In some parts I reflected on modalities and opportunities for speech interaction.

When characterizing mobile interaction I pointed out the following points:

  • Interacting while on the move
  • Interaction is one of the user’s tasks (besides others – e.g. walking, standing in a crowd)
  • Environment in which the interaction takes place changes (e.g. on the train with varying light and noise conditions)
  • Interruptions happen frequently (e.g. boarding the bus, crossing the road)
  • Application usage is short (typically seconds to minutes)

My small set of key issues and recommendations for mobile UIs is:

  • Simple and Understandable
  • Perceptive and Context-Aware
  • Unobtrusive, Embedded and Integrated
  • Low Cognitive Load and Peripheral Usage
  • Users want to be in Control (especially on the move)

The presentations and discussion at the workshop were very interesting and I got a number of ideas for multimodal user interfaces – including speech.

One issue that made me think more was the question about natural language speech vs. specific speech commands. A colleague pointed me to Speech Graffiti [1] / Universal Speech Interface at CMU. I wonder if it would make sense to invent a Human Computer Interaction language (with a simple grammar and a vocabulary) that we could teach in a course over several weeks (e.g. similar effort than touch typing on a QUERTY keyboard) or as a foreign language at school to have a new effective means for interaction. Could this make us more effictive in interacting with information? Or should we try harder to get natural languge interaction working? Looking at the way (experienced) people use Google we can see that people adapt very successfully – probably faster than systems improve…

From some of the talks it seems that “Push to talk” seems to be a real issue for users and a reason for many user related errors in speech systems. Users do not push at the appropriate time, especially when there are other tasks to do, and hence utterances are cut off at the start and end. I would guess continuous recording of the speech and using the “push to talk” only as an indicator where to search in the audio stream may be a solution.

[1] Tomko, S. and Rosenfeld, R. 2004. Speech graffiti vs. natural language: assessing the user experience. In Proceedings of HLT-NAACL 2004: Short Papers (Boston, Massachusetts, May 02 – 07, 2004). Human Language Technology Conference. Association for Computational Linguistics, Morristown, NJ, 73-76. http://www.cs.cmu.edu/~usi/papers/HLT04.pdf