Reading List: Developing Ubiquitous Computing Devices

 

Together with Thomas Kubitza I was teaching a class in the UBI summer school on Developing Ubiquitous Computing Devices. The summer school was held in Oulu and organized by Timo Ojala.

In total the summer school include the following 4 courses:

  • EXPERIENCE-DRIVEN DESIGN OF UBIQUITOUS INTERACTIONS IN URBAN SPACES Prof. Kaisa Väänänen-Vainio-Mattila, Tampere University of Technology, Finland & Dr. Jonna Häkkilä, University of Oulu, Finland
  • DESIGNING MOBILE AUGMENTED REALITY INTERFACES Prof. Mark Billinghurst, University of Canterbury, New Zealand
  • DEVELOPING UBIQUITOUS COMPUTING DEVICES Prof. Albrecht Schmidt, University of Stuttgart, Germany
  • URBAN RESOURCE NETWORKS Prof. Malcolm McCullough, University of Michigan, USA

There was more than work… if you are curious have a look at flickr for photos and more photos.

As some people asked for the reading list for our course on Developing Ubiquitous Computing Devices, I thought I post it here…. The reading list is also available as PDF for download.

The reading list comprises 4 areas that are relevant to our course. We expect that you have come across the original paper by Marc Weiser, introducing the concept of ubiquitous computing [1].

In the first part we have included papers that provide an overview of interaction concepts that are relevant in the context of ubiquitous computing. In particular this is tangible interaction [2a] [2b], reality based interaction [3], embedded interaction [4]. The concept of informative art [5] is introduced as well as the notion of persuasive technologies [16].This part is concluded with an overview of interaction with computers in the 21st century [6].

In the second part we have included a paper on how to create smart devices [7], which gives an overview of sensors that may be useful for creating novel and reactive devices. In [8] sensing is extended to context and context-awareness. In the third part we introduce the .NET Gadgeteer platform [9] and show some trends in the development of ubiquitous computing devices: how can we create new products once we can fabricate things [10] and enclosures [10b] and how ubicomp technologies enable new devices and devices concepts [11].

The final part provides some ideas for application scenarios that we plan to assess during the course. In [12] a concept of how to change a bed into a communication media is presented and in [13] a social alarm clock is presented. A recent study [14] shows the impact of technology on communication and in [15] an overview of novel alarm clocks and sleep monitoring devices is given.

References
[1] Weiser, M. (1991). The computer for the 21st century. Scientific american,265(3), 94-104. http://wiki.daimi.au.dk/pca/_files/weiser-orig.pdf
[2a] Ishii, H., & Ullmer, B. (1997, March). Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (pp. 234-241). ACM. http://doi.acm.org/10.1145/258549.258715 http://labs.rightnow.com/colloquium/papers/tangiblebits.pdf
[2b] Ishii, H. (2008, February). Tangible bits: beyond pixels. In Proceedings of the 2nd international conference on Tangible and embedded interaction (pp. xv-xxv). ACM. http://doi.acm.org/10.1145/1347390.1347392
[3] Jacob, R. J., Girouard, A., Hirshfield, L. M., Horn, M. S., Shaer, O., Solovey, E. T., & Zigelbaum, J. (2008, April). Reality-based interaction: a framework for post-WIMP interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 201-210). ACM. http://doi.acm.org/10.1145/1357054.1357089 http://research.cs.queensu.ca/~audrey/papers/chi08.pdf
[4] Kranz, M., Holleis, P., & Schmidt, A. (2010). Embedded interaction: Interacting with the internet of things. Internet Computing, IEEE, 14(2), 46-53. http://dx.doi.org/10.1109/MIC.2009.141 http://pure.ltu.se/portal/files/39756776/FINAL_PRINT_w2iot_preprint.pdf
[5] Ferscha, A. (2007). Informative art display metaphors. In Universal Access in Human-Computer Interaction. Ambient Interaction (pp. 82-92). Springer Berlin Heidelberg. http://www.pervasive.jku.at/Research/Publications/_Documents/InformativeArtDisplayMetaphors-ferscha2007.pdf
[6] Schmidt, A., Pfleging, B., Alt, F., Sahami, A., & Fitzpatrick, G. (2012). Interacting with 21st-Century Computers. Pervasive Computing, IEEE, 11(1), 22-31. http://www.hcilab.org/wp-content/uploads/schmidt-ieeepc-21century.pdf http://dx.doi.org/10.1109/MPRV.2011.81
[7] Schmidt, A., & Van Laerhoven, K. (2001). How to build smart appliances?.Personal Communications, IEEE, 8(4), 66-71. http://www.comp.lancs.ac.uk/~albrecht/pubs/pdf/schmidt_ieee_pc_08-2001.pdf
[8] Schmidt, A. (2013). Context-Aware Computing: Context-Awareness, Context-Aware User Interfaces, and Implicit Interaction. The Encyclopedia of Human-Computer Interaction, 2nd Ed. http://www.interaction-design.org/encyclopedia/context-aware_computing.html
[9] Villar, N., Scott, J., Hodges, S., Hammil, K., & Miller, C. (2012). . NET gadgeteer: a platform for custom devices. In Pervasive Computing (pp. 216-233). Springer Berlin Heidelberg. http://research.microsoft.com/pubs/163162/Gadgeteer%20Pervasive%202012%20Proof.pdf
[10] Schmidt, A., Doring, T., & Sylvester, A. (2011). Changing How We Make and Deliver Smart Devices: When Can I Print Out My New Phone?. Pervasive Computing, IEEE, 10(4), 6-9. http://www.hcilab.org/wp-content/uploads/2012/04/schmidt2011changing.pdf http://dx.doi.org/10.1109/MPRV.2011.68
[10b] Weichel C., Lau M., Gellersen,H. (2013). Enclosed: A Component-Centric Interface for Designing Prototype Enclosures. Tangible, embedded, and embodied interaction conference (TEI 2013) http://doi.acm.org/10.1145/2460625.2460659 http://www.csweichel.de/papers/2013-enclosed.pdf
[11] Hodges, S., Villar, N., Scott, J., & Schmidt, A. (2012). A New Era for Ubicomp Development. Pervasive Computing, IEEE, 11(1), 5-9. http://dx.doi.org/10.1109/MPRV.2012.1 http://research.microsoft.com/pubs/163175/ANewEraForUbiCompDevelopment-IEEEPervasiveComputing.pdf
[12] Dodge, C. (1997, March). The bed: a medium for intimate communication. InCHI’97 extended abstracts on Human factors in computing systems: looking to the future (pp. 371-372). ACM. http://doi.acm.org/10.1145/1120212.1120439
[13] Schmidt, A., Shirazi, A. S., & van Laerhoven, K. (2012). Are You in Bed with Technology?. Pervasive Computing, IEEE, 11(4), 4-7. http://dx.doi.org/10.1109/MPRV.2012.63
[14] Schmidt, A. (2006). Network alarm clock (The 3AD International Design Competition). Personal and Ubiquitous Computing, 10(2-3), 191-192. http://dx.doi.org/10.1007/s00779-005-0022-y http://old.hcilab.org/documents/Schmidt_NetworkAlarmClock.pdf
[15] Shirazi, A. S., Clawson, J., Hassanpour, Y., Tourian, M. J., Schmidt, A., Chi, E. H., Borazio, M., & Van Laerhoven, K. (2013). Already Up? Using Mobile Phones to Track & Share Sleep Behavior. International Journal of Human-Computer Studies. http://www.sciencedirect.com/science/article/pii/S1071581913000244
[16] Fogg, B. J. (2009, April). A behavior model for persuasive design. In Proceedings of the 4th international conference on persuasive technology (p. 40). ACM. http://bjfogg.com/fbm_files/page4_1.pdf

Appendix: .NET Gadgeteer Links (optional)

Visiting the Culture Lab in Newcastle

While being in the north of England I stopped by in Newcastle at the Culture Lab. If the CHI-conference is a measure for quality in research in Human Computer Interaction Culture Lab is currently one of the places to be – if you are not convinced have look at Patrick Olivier’s publications. The lab is one of a few places where I think a real ubicomp spirit is left – people developing new hardware and devices (e.g. mini data acquisition boards, specific wireless sensor, embedded actuators) and interdisciplinary research plays a central role. This is very refreshing to see, especially as so many others in Ubicomp have moved to mainly creating software on phones and tables…

Diana, one of our former students from Duisburg-Essen, is currently working on her master thesis in Newcastle. She looks into new tangible forms of interaction on table top UIs. Especially actuation of controls is a central question. The approach she uses for moving things is compared to other approached, e.g. [1], very simple but effective – looking forward to reading the paper on the technical details (I promised not to tell any details here). The example application she has developed is in chemistry education.

Some years back at a visit to the culture lab I had already seen some of the concepts and ideas for the kitchen. Over the last years this has progressed and the current state is very appealing. I really thing the screens behind glass in the black design make a huge difference. Using a set of small sensors they have implemented a set of aware kitchen utensils [2]. Matthias Kranz (back in our group in Munich) worked on a similar idea and created a knife that knows what it cuts [3]. It seems worthwhile to exploring the aware artifacts vision further …

References
[1] Gian Pangaro, Dan Maynes-Aminzade, and Hiroshi Ishii. 2002. The actuated workbench: computer-controlled actuation in tabletop tangible interfaces. In Proceedings of the 15th annual ACM symposium on User interface software and technology (UIST ’02). ACM, New York, NY, USA, 181-190. DOI=10.1145/571985.572011 http://doi.acm.org/10.1145/571985.572011 

[2] Wagner, J., Ploetz, T., Halteren, A. V., Hoonhout, J., Moynihan, P., Jackson, D., Ladha, C., et al. (2011). Towards a Pervasive Kitchen Infrastructure for Measuring Cooking Competence. Proc Int Conf Pervasive Computing Technologies for Healthcare (pp. 107-114). PDF

[3] Matthias Kranz, Albrecht Schmidt, Alexis Maldonado, Radu Bogdan Rusu, Michael Beetz, Benedikt Hörnler, and Gerhard Rigoll. 2007. Context-aware kitchen utilities. InProceedings of the 1st international conference on Tangible and embedded interaction (TEI ’07). ACM, New York, NY, USA, 213-214. DOI=10.1145/1226969.1227013 http://doi.acm.org/10.1145/1226969.1227013 (PDF)

Talk at demola, Finnish Ubicomp program

Jari Ikonen from the Finnish Ubicomp program contacted me last week – interestingly because I shared on this blog the information that I will be in Tampere – and it worked out that we met.

He showed and explained me the demola approach. I find this concept of teaching, training and innovation very exciting. In short demola offers space for students to work on challenging problems that are real and creates opportunities opportunities. Basically companies offer tasks/project to works on. Teams of students (potentially from different universities and fields) will work together to solve it as part of their studies – but the students also will own the IPR. I think that creates interesting teams in realistic settings and has probably a great potential for creating start-ups. Perhaps we should look at this model closer and see how we could create something similar…

As always when meeting interesting people time was too short… I gave an ad-hoc talk based on previous slides on “Mobile & Ubiquitous Computing and Beyond: Mobile Communication changed the world – what else do we need?” and we had a short but very interesting discussion.

Ubiquitous Computing – Ever wondered if we are there yet?

Given the technologies around us I sometimes wonder how close we are to a vision of ubiquitous computing. In this month IEEE Computer Invisible Computing column I had the pleasure to ask this question and share my view on it.

The short answer is: many technologies are ubiquitous but there is a lot more to come. In particular we see that many technologies (public displays, people centric sensing, and personal memory devices) are just around the corner and that they may have a large impact on how we perceive computing. For the long answer have a look at my article: ubiquitous computing – are we there yet? [1]. I have taken over responsibility for the invisible computing column from Bill Schilit who introduced the Invisible Computing column in 2003 [2].

Some years ago in 2006 Yvonne Rogers presented her view on how Ubicomp is going forward [3] contrasting it to Weiser’s Vision of calm computing. In her paper she introduces an alternative agenda that argue that we should engage people by ubicomp technologies rather than to make life easy, convenient and calm. Yvonne’s paper is an interesting starting point for getting students into this topic.

[1] Schmidt, A. 2010. Ubiquitous Computing: Are We There Yet? Computer 43, 2 (Feb. 2010), 95-97. DOI= http://dx.doi.org/10.1109/MC.2010.54

[2] Schilit, B. N. 2003. Mega-Utilities Drive Invisible Technologies. Computer 36, 2 (Feb. 2003), 97-99. DOI= http://dx.doi.org/10.1109/MC.2003.1178056

[3] Yvonne Rogers: Moving on from Weiser’s Vision of Calm Computing: Engaging UbiComp Experiences. Ubicomp 2006: 404-421

Some Interesting Papers and random Photos from Pervasive 2009

Pervasive 2009 had a really exciting program and provided a good overview of current research in pervasive and ubiquitous computing. Have a look at the proceedings of the pervasive 2009 conference. The Noh theater in Nara was a very special and enjoyable venue and it was organized perfectly – as one would expect when travelling to Japan.

The idea of having short and long papers together in the main track worked very well in my view. The number of demos and posters was much higher than in the years before – and that was great and very inspiring. Have a look at the photos for some of the posters and demos.
The program consisted of 20 full papers (18 pages) and 7 notes (8 pages) which were selected in a peer review process out of 147 submissions (113 full papers, 34 notes) which is a acceptance rate of 18%.

John Krumm presented his paper Realistic driving tips for location privacy – again having a good idea making the presentation interesting beyond its content (having review snippets in the footer of the slides – including a fake review). The paper explores the difficulties that arise when creating fake GPS tracks. He motivated that the probabilities need to be taken into account (e.g. you are usually on a road). I liked the approach and the paper is worthwhile to read. I think it could be interesting to compare the approach is not create the tracks but just share them between users (e.g. other people can use parts of my track as fake track and in return I get some tracks that I can use as fake tracks). http://dx.doi.org/10.1007/978-3-642-01516-8_4

If you phone knows where you are you can use this information to control your heating system. This was the basic idea of the research presented by Stephen Intille. They explored using GPS location of the users to automate control of the heating / air condition control in a house. It seems there is quite some potential for saving energy with technology typically used in the US (one temperature control for the whole house). In Europe where heating systems typically offer finer control (e.g. room level) the potential is probably larger.
http://dx.doi.org/10.1007/978-3-642-01516-8_8

James Scott presented a paper that showed how you can use force gestures to interact with a device. In contrast to previous research (e.g. GUMMI) the approach works with a ridged device and could be used with current screen technologies.
http://dx.doi.org/10.1007/978-3-642-01516-8_10

What do you need to figure out who is holding and using the remote control? This question is addressed in the paper “Inferring Identity Using Accelerometers in Television Remote Controls” that was presented by Jeff Hightower. They looked at how well button press sequences and accelerometer data give you information about which person is using the device.
http://dx.doi.org/10.1007/978-3-642-01516-8_11

Geo-fencing: confining Wi-Fi Coverage to Physical Boundaries is an example of how to create technological solutions to fit a user’s conceptual model of the world. As people have experience with the physical world and they have mechanisms to negotiate and use space and hence linking technologies that have typically other characteristics (e.g. wireless radio coverage) to the known concept is really interesting.
http://dx.doi.org/10.1007/978-3-642-01516-8_19

Situvis, a tool for visualizing sensor data, was presented by Adrian Clear from Aaron’s group in Dublin. The software, papers and a video is available at: http://situvis.com/. The basic idea is to have a parallel coordinate visualization of the different sensor information and to provide interaction mechanisms with the data.
http://dx.doi.org/10.1007/978-3-642-01516-8_22

Nathan Eagle presented the paper “Methodologies for continuous cellular tower data analysis”. He talked about the opportunities that arise when we have massive amounts of information from users – e.g. tracks from 200 million mobile phone user. It really is interesting that based on such methods we may get completely new insights into human behavior and social processes.
http://dx.doi.org/10.1007/978-3-642-01516-8_23

If you have seen a further interesting paper in the conference (and there are surely some) that I have missed feel free to give a link to them in the comments to this post.

Closing Panel at Ubicomp 2008

The closing panel at Ubicomp dicussed the last 10 years of ubicomp and potential future direction (with regard to community and technology). On the panel were Gregory Abowd, Hide Tokuda, Lars Eric Holmquist, Eric Paulos, and Albrecht Schmidt. In the following I will just describe some of the points I raised in my short statement. 
For me the first observation is that many ideas that were discussed at the first HUC99 (the first ubicomp conference started by Hans Gellsersen) – and were considered very speculative ideas have become common products and services by now (e.g. pocket bargain finder, predictive text input, mobile collaboration tool, mobile photo-sharing, location aware technologies). Here it is apparent that with regard to envisioning applications the conference has made impact. But there is a curious phenomenon: at the moment a device or service is available in the shop we do not recognize it as ubicomp anymore.
In some areas the complexity of the problems (when moving from the lab to the real world) has been underestimated – here context and context-awareness is a good example. If you realize the full vision it is basically solving AI. But nevertheless we progressed – there are applications for commercial mobile devices that do context and activity recognition – and it is just 10 years that we discussed this in a HUC99 in our paper on advanced interaction in context [1] – which was at that time really innovative! Context-awareness will happen – be patient 🙂 but its has to take into account: humans are adaptive, too. 
The papers which had a large impact (based on citations e.g. check [1] on google scholar) seem more the papers that score high on novelty, even if the may lack scientific rigor.
 
For future directions I hinted some general directions (not necessarily my research directions): 
  • Implanted activity recognition and interaction (put the sensing and actuation into the body solves a lot of the problems … obviously it creates many new ones, too) 
  • Implantable persuasion and amplifying bodily experiences. Here I gave the example that we would be able to create a device to motivate you do sports by making your back hurt. I used this to emphasise that ethics will play a large role in the future…
  • Prediction technologies (e.g. the weather forecast as an inspiration, forecasting traffic conditions, parking situation, restaurant business, costs, …) we will create systems that allo us to look up predictions (cost, quality of the experience, stress, time needed, etc.) for future activities (e.g. when choosing a restaurants, booking a travel, deciding on dating a person, making a business deal, accepting a position, …)
  • And finally I suggested that we will have fun with papers on privacy published now when reading them in 20 years 🙂 because our perception of this topic will change massively.
With regard to the community I made the statement that Ubicomp became the Starbucks of ubiquitous computing research – premium but based on the US idea of quality. Have you ever been in a Vienna coffee house, in an Italian espresso bar, or had tea in the middle east – it is very different. We lost some of the international spirit and we stopped arguing what good research in ubicomp is – this discurse should be started again!. Looking at the countries where ubicomp technologies come from (e.g. a lot from Aisia and Europe) we should again make a effort to more value the international diversity and the different styles and approaches in ubicomp research – scientific rigor is not the only axis to consider. 
[1] Schmidt, A., Aidoo, K. A., Takaluoma, A., Tuomela, U., Laerhoven, K. V., and Velde, W. V. 1999. Advanced Interaction in Context. In Proceedings of the 1st international Symposium on Handheld and Ubiquitous Computing (Karlsruhe, Germany, September 27 – 29, 1999). H. Gellersen, Ed. Lecture Notes In Computer Science, vol. 1707. Springer-Verlag, London, 89-101. DOI= http://dx.doi.org/10.1007/3-540-48157-5_10

Keynote at Ubicomp 2008, Dr. Shim Yoon

The opening keynote at Ubicomp 2008 was presented by Dr. Shim Yoon (Vice President Samsung SDS) on Realizing Ubiquitous Cities. She stated out with a few  scenarios of life in the future. Information is embedded into the environment – just in place and just in time – just what you need. Environments become pleasant and technologies look nice, so you can mistake a power distribution box for a sculpture. Cities will be really save as monitoring will be ubiquitous and continuous. Much of the vision is convincing and convenient (given that you have given up on privacy, too). 
There has been an interesting slide in the scenarios that  made me smile (in short “22:47, pick up the book reserved in advance for lending after taking a walk”). It basically says we will still have pretty long days and books will be around, too. 

I think in the age of ebooks I will not pick up books late in the evening. Looking at the analogies of previous technical revolutions one can argue that these reduced the number of hours people had to work to have a good life – it seems this technical revolution creates the opposite (u-work everywhere and anytime 😉 – is fine with me because I like my work :-)She described how technologies changes the urban culture, e.g. provision of water, later roads and harbors, and very recent fiber optics and mobile communications. In her view ubiquitous technologies will add a further paradigm shift. 

The overall concept of u-city can be condensed to her statement:  “u-city can solve complicated urban problems” and this is attempted by combining political, physical and techno solutions. It is envisioned having IT included in all city elements. The approach follows a layered design with a base layer of the communication infrastructure (e.g. fixed and wireless infrastructure). In the middle there are the u-facility arrangements (e.g. sensor network, CCTV, power distribution), and on top u-services (u-home, u-office, u-traffic, …) are provided. 


It is envisioned that u-services are partly free (provided by the government) and partly
 commercial. Some of the examples for services see straight forward and very close (e.g. u-urban facility management that allows to monitor, check, maintain the infrastructure) and some are somehow scary (e.g. u-safety and security allows complete monitoring over the while city, ranging from fire recognition to behavioral analysis of its inhabitants). She pointed out that u-service modeling needs to include technical aspects as well as a business analysis.
The importance Korea sees in this topic is amazing and is summarized by the following quote “U-city is a key aspect in the next generation industry and economy in Korea”. The Korean government is behind the developments and urban planning embraces the concept and puts it at the center of the planning. Looking at the u-urban information concept it becomes apparent that this concept is valuable along the entire lifecycle, including planning, simulation, management and operation. One catch is that it has to be integrated while building, otherwise it will be expensive and it is not possible to gain the full potential. This is the bad message for most developed countries as we usually change our cities and do not build them from scratch…

Explaining more on the realization of u-city she argued that a new process is required and that construction and IT processes have  to go together. During city planning a u-city consulting process is performed (creating an IT master plan for the new city including the envisioning of the services). In the building process the IT-infrastructure is included (when it is easiest to realize). Running and managing the city includes operating U-city (basically running the IT services by a specific UbiCenter – a control center for the city).

In the final part of her keynote she talked a little about the Samsung U-City strategy. Here she saw a necessity to merge the city construction with the provision of IT capabilities. This ranges from  developing convergence Services (e.g. ubiquitous communication service, city control an & security services) to the creation of comprehensive services for complex housing environments. An example of a infrastructure device she mentioned is a complex lighting pole (which could include information display, LED lighting, CCTV camera, wireless access points, etc.). In summary the strategy is the development (and potentially the export) of world class u-city technologies. 
In the question session I raised the issue if this may lead to even more people moving from the rural areas to cities because of the new qualities such u-cities will provide. And it seems there will be even a greater difference between rural areas and cities in the future and this may strengthen the trend towards mega-cities. Perhaps we should think a bit more about u-rural…

Visit at Microsoft in Redmond

AJ Brush and John Krumm organize for the people who are in Redmond for the Ubicomp PC meeting a visit to Microsoft. In the morning we got a tour at the home lab – Microsoft’s vision of future home environments – was quite interesting, but had to sign an NDA.
After lunch we went over to Microsoft Research (which is in a new building). We got to see some cool demos. Andy Wilson showed us some new stuff moving the SURFACE forward (physics rocks!). I learned more about depth sensing cameras and Andy showed a fun application [1] – there is video about it, too. Patrick Baudisch talked us through the ideas of LucidTouch [2] and more general about future interaction with small mobile devices. The idea of using the finger behind the screen and the means to increase the precision has many interesting aspects. I found the set of people that work at MSR as impressive as the demos – it seems to be a really exciting work environment.

The atrium of the new building is amazing for playing Frisbee and shoot rubber band missiles. And waiting for the pizza with those toys around proved yet again that researchers are often like kids 😉

[1] Wilson, A. Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction. Tabletop 2007: The 2nd IEEE International Workshop on Horizontal Interactive Human-Computer Systems, 2007.

[2] Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., Shen, C. LucidTouch: A See-Through Mobile Device. In Proceedings of UIST 2007, Newport, Rhode Island, October 7-10, 2007, pp. 269–278 http://www.patrickbaudisch.com/projects/lucidtouch/

Ali joined our group

Last month Aliresa Sahami finished his master thesis on multi-tactile interaction at BIT Bonn and joined our group in Essen. Ali worked for me a student resesearch assistant at Fraunhofer IAIS. During his studies in Bonn we published an interesting workshop paper on mobile health [1] and gave a related demo at Ubicomp [2].

[1] Alt, F., Sahami Shirazi, A., Schmidt, A. Monitoring Heartbeat per Day to Motivate Increasing Physical Activity. Ubiwell workshop at Ubicomp 2007.

[2] Sahami Shirazi, A.; Cheng, D.; Kroell, O.; Kern, D.; Schmidt, A.: CardioViz: Contextual Capture and Visualization for Long-term ECG Data. In: Adjunct Proceedings of Ubicomp 2007.

Where are my things – Would Smart-its friends help in a real world scenario?

At social events of conferences interesting things happen. One issue with a borrowed key reminded me of a paper that colleagues in the smart-its project wrote several years ago – smart-its friends [1]. The central idea was to have means to connect objects (make them friends) by a gesture interaction, which is detected by comparing acceleration values. Technically it is feasible and highly interesting, but I wonder about the real world applicability – but the missing key may be evidence for it…

[1] Holmquist, L. E., Mattern, F., Schiele, B., Alahuhta, P., Beigl, M., and Gellersen, H. 2001. Smart-Its Friends: A Technique for Users to Easily Establish Connections between Smart Artefacts. In Proceedings of the 3rd international Conference on Ubiquitous Computing (Atlanta, Georgia, USA, September 30 – October 02, 2001). G. D. Abowd, B. Brumitt, and S. A. Shafer, Eds. Lecture Notes In Computer Science, vol. 2201. Springer-Verlag, London, 116-122.

Reminded of the Ubicomp Vision

Today I was reminded of a discussion in 1998 on the implications of computing technologies becoming cheaper and cheaper. Even then it seemed inevitable that many artifacts will include computational and perceptual qualities. The discussion was in the context of the European project TEA (technology for enabling awareness) where we built a context-aware phone [1]. Walter van de Velde suggested imagining that processors, sensors, communication will only cost cents (or will be virtually free as part of the production process) and we worked on the question: what products and services will emerge? One generic answer then was than any product of a value 20$ and above will include computing and sensing capabilities, if there is any (even a minimal) advantage achieved by this.

Michael Beigl made it more concrete and found coffee mugs (which were more than 20$ each) and attached a processor, communication and sensors. The MediaCup [2] showed several interesting results and underlined that such approach makes sense if there is an advantage.

Today I saw in an office of a former colleague in Munich two objects that had perceptual qualities and output (not really processing yet). One object is a plastic toad that makes a noise when you move and the other is a rubber pig that makes a noise when you open the fridge (reacts on change in level, but did not work). This made me wonder if we were only partially right – yes objects will have sensors included, yes there will be processing, but no there is no need that it makes sense. Or perhaps having it as a gadget is advantage enough…

[1] Schmidt, A., Aidoo, K. A., Takaluoma, A., Tuomela, U., Laerhoven, K. V., and Velde, W. V. 1999. Advanced Interaction in Context. In Proceedings of the 1st international Symposium on Handheld and Ubiquitous Computing (Karlsruhe, Germany, September 27 – 29, 1999). H. Gellersen, Ed. Lecture Notes In Computer Science, vol. 1707. Springer-Verlag, London, 89-101. DOI= http://dx.doi.org/10.1007/3-540-48157-5_10

[2] Gellersen, H. W., Schmidt, A., and Beigl, M. 2002. Multi-sensor context-awareness in mobile devices and smart artifacts. Mob. Netw. Appl. 7, 5 (Oct. 2002), 341-351. DOI= http://dx.doi.org/10.1023/A:1016587515822

Talk at the expert meeting on RFID and ubicomp

In Frankfurt there was today an expert meeting on RFID and ubicomp organized by the Fraunhofer ISI. The purpose was a discussion about the impact of RFID technologies. The organizers will use our input to inform the creation of a document of technology assessment for the German parliament. The majority of the participants came from companies developing RFID technology or system.
In the first part of my talk “RFID and Beyond” I highlighted results from two workshops where I was a co-organizer: PTA2006 and Pertec2007 held at the Pervasive and Percom conferences. The results were also published in 2 papers in the IEEE Pervasive Computing magazine, see [1] and [2]. After this I showed some future visions and scenarios, namely the Smart-Its & MediaCup (foto from Birgit at Teco) [3], the SensorKnife [4], and the aware goods project [5]. Michael Müller extended the idea of the first aware goods project with a mobile phone based prototype – which we still have not written up for publication.

For me the technology assessment in Germany seems still often very much centred on threats and looks much less at opportunities. Looking at developments in Asia and in particular in Korea (e.g. U-City) I hope politics in Germany will in the future more often see the positive sides, too. Technology assessment can become a means to find opportunities and ideas to support innovation. For me it seems that a lot of the risks people attribute to RFID are not based on scientific results – is appears rather media induced… Positive cases such as wireless key systems and transport tickets (basically RFID technology) are in widespread use without much problems and great value for users – but not present in the public discussion.

One interesting estimated was that about 200 parts of the several thousands (e.g. safety related parts, large parts, parts that are often stolen, expensive parts) per car will tagged with RFID in the next 10 years to ease logistics, production and maintenance.

[1] Schmidt, A.; Spiekermann, S.; Gershman, A.; Michahelles, F., “Real-World Challenges of Pervasive Computing“, Pervasive Computing, IEEE , vol.5, no.3pp. 91- 93, c3, July-Sept. 2006.

[2] Michahelles, F.; Thiesse, F.; Schmidt, A.; Williams, J. R.: Pervasive RFID and Near Field Communication Technology. In: IEEE Pervasive Computing, vol. 6, no. 3, pp. 94-96, c3, Jul., 2007.

[3] Hans-Werner Gellersen, Albrecht Schmidt, Michael Beigl: Multi-Sensor Context-Awareness in Mobile Devices and Smart Artifacts. MONET 7(5): 341-351 (2002)

[4] Matthias Kranz, Albrecht Schmidt, Alexis Maldonado, Radu Bogdan Rusu, Michael Beetz, Benedikt Hörnler, Gerhard Rigoll: Context-aware kitchen utilities. Tangible and Embedded Interaction 2007: 213-214

[5] Anke Thede, Albrecht Schmidt, Christian Merz: Integration of Goods Delivery Supervision into E-commerce Supply Chain. WELCOM 2001: 206-218

Interactive Mirrors – an upcoming topic beyond the idea?

At the CHI PC-meeting in Amsterdam I talked to a number of people – and it seems we are not the only one’s who are interested in interactive mirrors. It seems that breaking the physical limitations in time and space, a convention mirror imposes, creates some interest within the research community.

I talked to Boris de Ruyter about the Philips mirror project in the homelab and learned from Bo Begole about their work on interactive mirrors at PARC. It may be interesting to propose a workshop on interactive mirrors at one of the upcoming conference to get the people together looking into this topic.
In the hotel in Schiphol there was a mirror display for adverts. It did not really link any functionality of the mirror with the display, but nevertheless it was a aesthetically pleasing installation

Talk at the opening of the Fraunhofer IAO interaction lab

The Fraunhofer institute IAO opened today a new interaction lab in Stuttgart under the topic interaction with all senses. Prof. Spath, director of the Fraunhofer IAO, made a strong argument for new user interfaces. In his talk he discussed adaptive cruse control in cars as an example for user interface challenges.

My talk on “implicit interaction – smart living in smart environments” argues for a sensible mix of user centred design and technology driven innovation. As one example I used the Sensor-Knife which Matthias Kranz implemented.

Prof. Jürgen Ziegler, a colleague at the University of Duisburg-Essen who was previously at IAO, showed in his talk a short video of a “galvanic vestibular stimulation” GVS explored by NTT (SIGGRAPH 2005 Demo) to highlight trends and indicate at the same time ethical problems that can arise when we interfere with human senses.

Ubicomp 2007

Over the last few days at Ubicomp 2007 in Innsbruck it was great to catch up with many people from the community. The discussions in the evenings are very inspiring and so were some of the talks.

We tried to explain the idea of ubiquitous computing to journalist and it seems they got the idea. And hence Ubicomp 2007 was featured in the Austrian Press:

Ubicomp 2008 will be in Korea.

CardioViz Demo at Ubicomp 2007

Alireza Sahami presented our CardiViz project at the demo session at Ubicomp. We were very happy that the project that was the result of our IPEC course on developing mobile applications was accepted as a demo.

For more details see:
Alireza Sahami Shirazi, Diana Cheng, Oliver Kroell, Dagmar Kern, Albrecht Schmidt. CardioViz: Contextual Capture and Visualization for Long-term ECG Data. Adjunct Proceedings of Ubicomp 2007 (Demo).

Jonna Häkkilä, Anind Dey, Kari Hjelt, and I organized organized the Ubiwell workshop (Interaction with Ubiquitous Wellness and Healthcare Applications) at this years pervasive. Alireza presented another paper on heartbeat monitoring there:
Florian Alt, Alireza Sahami Shirazi, Albrecht Schmidt. Monitoring Heartbeat per Day to Motivate Increasing Physical Activity. UbiWell workshop@Ubicomp 2007.

Ubiquitous, Pervasive and Ambient Computing – Clarification of Terms

In the resent month the question about ubiquitous, pervasive, ambient computing came up several times. An email by Jos Van Esbroeck motivated me to write my view on it…

Clarifying the terms seems an ongoing process as various communities and individuals use each of those terms for new things they are doing.

For me the best way to discriminate the terms ubiquitous computing, pervasive computing, and ambient intelligence is by their origin, history and research communities.

The first term (ubiquitous computing, ubicomp) is linked to Mark Weiser and his vision of computing in the 21st century [1]. In the research community its is very much connected to ubiquitous and pervasive systems that have the user somewhere in the loop. The ubicomp conference [2] seems more focused on user experience than on pure technology.

Pervasive Computing was pushed in the mid 1990s, more by industry and in particular by IBM. Pervasive computing seems from its origin more focused on technologies and solutions than on a particular vision. The two major conferences related to this topic: pervasive [3] and percom [4] are more systems and network focused, however always keeping some attention to the user experience perspective. Here, in particular with percom, many in the research community have their origin in the networking and distributed systems world. To me pervasive computing seems more technical than ubiquitous computing and includes systems that do no have direct human users involved.

The term ambient intelligence was introduced by the European funding agencies in the Framework 5 vision. Around the same time as the Philips Home-lab that drives the term, too. Here, similar to ubicomp, the vision of a new quality of user experience is a driving factor. The research that falls under this label by now is broad and I think it is very similar to the research in ubiquious computing. There is also a European conference on ambient intelligence [5].

Many people that are involved in ubicomp/pervasive/percom are also active in one more traditional research community. In particular many people are additionally involved in user interface research (e.g. CHI-Community), mobile computing and mobile systems, networking and distributed systems.

A very early topic related to the whole field is context-awareness as introduced by Schilit [6] who was working with Weiser. In my PhD dissertation I have looked more into the relationship between ubicomp and context-awareness – it has the title Ubiquitous Computing – Computing in Context [7]

In parallel subtopic in the above field have emerged that look at specific aspects, e.g. internet of things [8] (not necessarily a human in the loop), wearable computing (computing in cloth), smart environments (computing in buildings and furniture), tangible and embedded interaction [9] (looking at the interaction side), smart objects, … and probably many more.

There is also an interesting trend that many of the topics, if they are a bit matured, move back into the traditional communities.

[1] Mark Weiser. The Computer for the Twenty-First Century. Scientific American 265, 3 (September 1991), 94-104
[2] http://www.ubicomp.org/
[3] http://pervasive2008.org/
[4] http://www.percom.org/
[5] http://www.ami-07.org/
[6] B. Schilit, N. Adams, and R. Want. (1994). “Context-aware computing applications“. IEEE Workshop on Mobile Computing Systems and Applications (WMCSA’94), Santa Cruz, CA, US: 89-101 .
[7] Albrecht Schmidt(2003). “Ubiquitous Computing – Computing in Context“. PhD dissertation, Lancaster Univeristy.
[8] http://
www.internetofthings-2008.org
[9] http://www.tei-conf.org/

Wall-Sized Printed Adverts with Integrated Screen

At Zurich Airport Orange and Nokia are running a large printed advert. At a first glance it looks just as a printed large scale poster. The TV screen in one poster and the projected writing on top of another poster are seamlessly integrated. The media design of the overall installation is appealing.

The active screen (could be a 50 inch plasma TV) is the screen of the mobile phone and shows the navigation application. In contrast to most other installations, where screens and printed posters are used, this appears right and it catches people’s attention.

There is work from Scott Klemmer’s group at Stanford that looks the relationship between the printed displays and projection/displays for various applications. The Gigaprints project was shown as a video at Ubicomp 2006.

Large scale sensor network connected to public displays

The airport Köln-Bonn (CGN) has all the parking spaces monitored with a simple sensor (detects if there is a car or not) and provides displays at the entrance showing the number of open spaces and has active signage in the parking garage leading to the free spaces – additionally it is visualized above each space – probably more a maintenance functions to see if the sensor works.

(looking at the pictures I have probably parked on women-only parking spots…)

Public Displays – Making Life More Predictable

On my way home from Toronto it was surprising how many public displays I saw that provided me with ”information about the future”, e.g. telling me when I will be out of time to cross the road, when the next train is due or when my luggage will arrive. These kinds of predictions or contexts are simple to gather and easy to present and best of all: the human is in control and can act on the information. Overall it is reassuring even if the context information is wrong (this is another story about my luggage ;-).

Pervasive Computing and Ethics

Together with Boriana Koleva I organized the doctoral colloquium at Pervasive 2007 in Toronto. We had 9 students presenting and discussing their PhD work with us.

One central observation was that we come to a point where we have to make more and more ethical decisions. Many things that are technical feasible and harmless within the lab may have sincere implications in the real world. If technologies for tracking, tracing and mining (e.g. social network analysis, location based services, context-aware systems) are deployed beyond the lab the question of choice becomes a real issue – are users aware of it and can they opt-out?

In the area of context-awareness technology has moved on since I started my PhD on the topic nearly 10 years ago – but amazingly scenarios did change little. Automatically detecting a meeting is still on the students slides. The more I learn and understand about context-awareness the more it becomes apparent that this apparently simple use case is amazingly hard!

Why shopping is fun – thoughts on intelligent user interfaces or why n=1 is not enough

Some weeks ago I saw for the first time one of the intelligent scales in the wild (=outside the lab). At that time I was really impressed how well it worked (sample size: n=1, product: banana, pack: no-bag, recognition performance: 100%). Last time I was too late so there was no time to play or see other people using it – but today I had some 5 minutes to invest.

The basic idea of the scale is simple and quite convincing. The customers put their purchase on the scales. A camera makes a guess what it is and the selection menu is reduced to the candidates that match the guess of the camera. Additionally, there is always a button to get all the options (as in the old version without the camera). It appears that this should make things easier.

I observed people trying to weigh different fruits and vegetables in bags and without bags (obviously I tried it myself, too). It did not work very often but interestingly people did not care much. It looked as most people did not really realise that this is meant to be an intelligent user interface. They probably just wondered why the display is showing always different things, but as they are intelligent themselves they found a way to deal with.

Overall it seems that it does really well on bananas which are not wrapped in a bag (my initial test case) and does not too well on many other things. I think the scales are an interesting example of a invisible interface.

Overall this is again a remainder that user tests that are small may be utterly wrong.

Panel on Sensor Networks – Applications are the Key

Debora Estrin made an interesting statement. The “early challenges” (the thousands or millions of randomly scattered sensor notes) do not have much applicability outside the battlefield. The new challenges are heterogeneity (specific sensors with specific capabilities) and interactivity (basically sense-making is a process where humans are involved). She made the point that the logical consequence is that dealing with data is the essential issue and statistics have an increasing role. Furthermore these new research directions make a stromg call for application driven research. With these very insightful comments she criticised a lot of the current work in sensor networks. Especially the observation that there is no such thing as a “general sensor” – it points out that concrete applications are required to make meaningful contributions, even to basic research in sensor networks research.

Best Demo Award at Percom 2007

Gregor got for our Perci prototype (Supporting Mobile Service Usage through Physical Mobile Interaction) the best demo award. So it paid off that he spend a night configuring the data services on the phones for the US networks 😉 It is amazing that it is still quite an effort to configure data services for a new provider.

Talks and Demos at PerCom 2007 in White Plains, NY

This year we (my previous group from LMU Munich) have a significant presence at PerCom. Form the 20 full papers the Embedded Interaction Research Group (www.hcilab.org) has 3, and additionally 1 of the 7 concise papers is from us. With a total of 207 submissions and an acceptance rate of around 10% this is quite an achievement for the team – and a good high point for the project before moving it to University of Bonn.

Gregor(y) Broll had the demo developed in the Perci-Project yesterday. Lucia Terrenghi and I had our talks today. And Gregor Broll, Sebastian Boring, and Raphael Wimmer have their talks tomorrow. Overall the conference has quite a diverse and interesting program, which seem to be more technical and network oriented than Ubicomp or Pervasive. The publications will be online available at the IEEE digital library or on our new publication webpage.

What can you do with a Wii controller? Use tape and connect a toothbrush and program a nice UI (fish tank) in Flash. Quite an interesting demo from Waseda University in Japan – at least the person who did the demo had really clean teeth in the evening.


PerTec Workshop in NY, Ripping off the Antenna

At PerCom 2007 (www.percom.org) Florian Michahelles (Auto-ID labs, ETH Zurich), Frédéric Thiesse (University of St. Gallen), John R. Williams (MIT Cambridge) and I are running the the PerTec workshop (www.autoidlabs.org/events/pertec2007). There is quite some interest in the topic and the range of topics is from technical to user interface and security.

In contrast to the workshop 1 year ago at Pervasive 2006 it seems that item level RFID-tagging is undisputed and that the only discussion point is when it is coming – in 6 month or 10 years. There is also still some discussion about what types of products are the first ones that are tagged – is it pharmaceuticals or cloth?

There was an interesting contribution by Paul Moskowitz from IBM, the clipped tag. It is a tag where you can physically rip off some part the antenna to reduce the read range from several meters to centimetres (see the pictures). It is really interesting that people can do a very clear and visible action to change the characteristics of a tag. The only questions that remains for me – will people trust that this really rips of the antenna? Probably yes…

The topics we discussed included security, privacy, location and RFID, end user issues, and connection of sensors to RFID, we hope to write it up in an overview article.

In the break out groups one discussion centered on the question what would we need to enable end-users to create novel applications using RFID? Further results of the discussion will be available on the workshop webpage soon.

The overall theme that emerged again and again is impact of real world constraints in RFID systems. Questions like: Can you achieve anonymity with a certain protocol? Can not be answered without knowing how it is used in the real world. Especially having recently learned a lot about data-mining (being at Fraunhofer IAIS) for me the questions of exploiting data collected in RFID systems looks really challenging.

Wearable Computing – Is it here?

10 years after I have got seen first see the crazy idea of wearable computing it appears that the technology is now really pushing into the marked. Even though one could argue whether or not this is really wearable computing (but this argument is as old as the idea of wearable computing). The last thing I would have expected 10 years ago was Bavarian Lederhosen with a built in user interface for an mp3-player. But nevertheless many challenges are still the same (integration with the aesthetics and fashion statement, durability and wash-ability, connectivity between computer and garment, integrated user interfaces) and some are nicely solved (http://www.smarttextiles.net/).

After Falke presented some years ago a sensor/ECG-t-shirt there was another one at CeBIT this year. It is not on the marked yet and it did not yet look fully convincing (you need to button in the electrodes). In our lab class on programming mobile systems we could use such a t-shirt. Currently we use stick on electrodes from http://www.alivetec.com/ which work really good but for the scenarios in mind having a t-shirt would be nicer.

In my recent entry on St. Petersburg I wondered about a communication glove. There was one on display – it includes only the essential (basically a speaker and a microphone) – but it enforced my opinion that creating a communication glove would be an interesting project. There were also hats with included speaker and microphones.

I was told that garments that keep warm – with active heating – are a hot topic 😉 Not fully convinced, but if projected consequently into the future it could change the way we dress completely. Want to show off you body at -20°C? Just power up the heating in your underwear a bit more and walk outdoors in your favourite summer dress without a coat. Not sure if this is the way we should push, I rather take my warm coat and save some energy. Apropos saving energy – there were interesting laptop bags with solar panels on the outside for harvesting energy.

CeBIT: Context-Awareness for Novel Transport and Logistics Services

At the Fraunhofer Forum at CeBIT I had the chance to talk about future transport solutions. One of my students in Munich (Michael Müller) is working on a pro-active transport container. In the project aware-goods (http://www.comp.lancs.ac.uk/~albrecht/awaregoods/) we looked into this domain already in 2000 at TecO together with SAP Research. Reiterating over the idea is really interesting as now the technology is really here – sensor network nodes are available, mobile phones offer massive processing power, there is ubiquitous data connectivity, and web service interfaces are available for many applications.

The big question is now on algorithms and software for context recognition and data mining in the collected sensor data, the integration with processes and company software systems, and the exploitation for optimising transport and logistics processes.

Affectionate Computing

Thecla Schiphorst introduced us in her talk “PillowTalk: Can We Afford Intimacy? to the concept of Affectionate-Computing.
The central question is really how can we create intimacy in communication an interaction with and through technology? The prototype showed networked soft objects, that include sensors that recognize tactile qualities and gesture interaction. There are more details in her paper published at TEI’07.

Nigel visiting, discussion on transportation

Nigel Davies from Lancaster University in the UK is visiting our group at Fraunhofer and b-it. It was really good to have some time – basically most of the day – to discuss interesting research questions and new direction in pervasive computing.

We discovered that we recently have both looked more into issues related to transport and tracking. I learned about possibilities for developing apps for the TomTom navigation system(which is running linux) and the trakm8 in-car-unit. We had a look at Michael Müller’s (master student I supervise in Munich) video of the smart transport container (more details on this at CeBIT).

Both of us are really interested how personal transportation is going to change and how computer science will make its contributions to that? The discussion on motives and motivations for choosing a particular mode of transport, what are the perceived values of certain transport modalities, and how can we influence them made me really think of implications of pervasive computing that are to come.

It is remarkable that as “cars” become more programmable interesting research projects start at many different institutions looking into automotive computing on an application level with an aim for new user experience and additional services/functionalities.