This paper introduces the special issue of AI EDAM on the role of gesture in designing. It starts with the context of the papers submitted and a summary of the papers accepted. We then introduce gesture studies, one of the two main domains with which this Special Issue is concerned. We do not introduce design research: we suppose the readers of AI EDAM are familiar with this domain. After this general introduction to the domain of gesture studies, we provide an overview of gestures in design, that is, the research environment of the papers in this Special Issue. We then discuss some dimensions on which these papers differ, as well as some on which they are related.

This paper illustrates the complexity of pointing as it is employed in a design workshop. Using the method of interaction analysis, we argue that pointing is not merely employed to index, locate, or fix reference to an object. It also constitutes a practice for reestablishing intersubjectivity and solving interactional trouble such as misunderstandings or disagreements by virtue of enlisting something as part of the participants’ shared experience. We use this analysis to discuss implications for how such practices might be supported with computer mediation, arguing for a “bricolage” approach to systems development that emphasizes the provision of resources for users to collaboratively negotiate the accomplishment of intersubjectivity rather than systems that try to support pointing as a specific gestural action.

One challenge in building collaborative design tools that use speech and sketch input is distinguishing gesture pen strokes from those representing device structure, that is, object strokes. In previous work, we developed a gesture/object classifier that uses features computed from the pen strokes and the speech aligned with them. Experiments indicated that the speech features were the most important for distinguishing gestures, thus indicating the critical importance of the speech–sketch alignment. Consequently, we have developed a new alignment technique that employs a two-step process: the speech is first explicitly segmented (primarily into clauses), then the segments are aligned with the pen strokes. Our speech segmentation step is unique in that it uses sketch features for locating segment boundaries in multimodal dialog. In addition, it uses a single classifier to directly combine word-based, prosodic (pause), and sketch-based features. In the second step, segments are initially aligned with strokes based on temporal correlation, and then classifiers are used to detect and correct two common alignment errors. Our two-step technique has proven to be substantially more accurate at alignment than the existing technique that lacked explicit segmentation. It is more important that, for nearly all cases, our new technique results in greater gesture classification accuracy than the existing technique, and performed nearly as well as the benchmark manual speech–sketch alignment.

Gestures play an important role in communication. They support the listener, who is trying to understand the speaker. However, they also support the speaker by facilitating the conceptualization and verbalization of messages and reducing cognitive load. Gestures thus play an important role in collaboration and also in problem-solving tasks. In human–computer interaction, gestures are also used to facilitate communication with digital applications, because their expressive nature can enable less constraining and more intuitive digital interactions than conventional user interfaces. Although gesture research in the social sciences typically considers empty-handed gestures, digital gesture interactions often make use of hand-held objects or touch surfaces to capture gestures that would be difficult to track in free space. In most cases, the physical objects used to make these gestures serve primarily as a means of sensing or input. In contrast, tangible interaction makes use of physical objects as embodiments of digital information. The physical objects in a tangible interface thus serve as representations as well as controls for the digital information they are associated with. Building on this concept, gesture interaction has the potential to make use of the physical properties of hand-held objects to enhance or change the functionality of the gestures made. In this paper, we look at the design opportunities that arise at the intersection of gesture and tangible interaction. We believe that gesturing while holding physical artifacts opens up a new interaction design space for collaborative digital applications that is largely unexplored. We provide a survey of gesture interaction work as it relates to tangible and touch interaction. Based on this survey, we define the design space of tangible gesture interaction as the use of physical devices for facilitating, supporting, enhancing, or tracking gestures people make for digital interaction purposes, and outline the design opportunities in this space.

Function models are frequently used in engineering design to describe the technical functions that a product performs. This paper investigates the use of the functional basis, a function vocabulary developed to aid in communication and archiving of product function information, in describing consumer products that have been decomposed, analyzed, modeled functionally, and stored in a Web-based design repository. The frequency of use of function terms and phrases in 11 graphical and 110 list-based representations in the repository is examined and used to analyze the organization and expressiveness of the functional basis and function models. Within the context of reverse engineering, we determined that the modeling resolution provided by the hierarchical levels, especially the tertiary level, is inadequate for function modeling; the tertiary terms are inappropriate for capturing sufficient details desired by modelers for archiving and reuse, and there is a need for a more expressive flow terms and flow qualifiers in the vocabulary. A critical comparison is also presented of two representations in the design repository: function structures and function lists. The conclusions are used to identify new research opportunities, including the extension of the vocabulary to incorporate flow qualifiers in addition to more expressive terms.

Feature technology is considered an essential tool for integrating design and manufacturing. Automatic feature recognition (AFR) has provided the greatest contribution to fully automated computer-aided process planning system development. The objective of this paper is to review approaches based on application of artificial neural networks for solving major AFR problems. The analysis presented in this paper shows which approaches are suitable for different individual applications and how far away we are from the formation of a general AFR algorithm.