Measuring the performance of search systems is essential in improving their effectiveness. Computing measures (or metrics, used synonymously in this chapter) lets search providers benchmark current performance, as well as quantify the impact of any changes. Some measures target the outcomes of the search process (e.g., the relevance of the found items), while some are more focused on the search process itself (e.g., the efficiency or cognitive load of the search process). Although numerous measures of search system performance have been proposed, none can fully evaluate search systems from all perspectives. As search systems become more sophisticated and support a broader range of tasks, new evaluation metrics and metric combinations will be needed.

Engelbart (1962, p. 1) suggested that the increased capability attributable to augmenting human intellect would likely lead to: “more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that could not previously be solved.” Systems that offer such opportunities cannot simply be evaluated using traditional retrieval measures such as precision and recall, which only consider the relevance of the found content and how much of the relevant content is found. In this case, we would need metrics that assess the quality of the solution and assess the impact of the search process on people's understanding of the subject matter.

There are two groups of metrics considered in this chapter: (1) those that assess the search process in which the searcher was engaged; and (2) those that target the outcomes attained as a result of that process. For completeness, I cover some of the traditional metrics, but many of those discussed draw on research in other communities, such as psychology. Irrespective of the target for the metric computation, with enhancements in next-generation search systems, evaluation metrics (and methods, discussed in the Chapter 11) need to cater to a diverse range of searchers, tasks, and interactivity.

Traditionally, the unit of retrieval evaluation is the search query. Next-generation search systems, however, place an emphasis on supporting the completion of complete search tasks end-to-end (rather than considering queries independently and satisfying task-relevant information needs one query at a time).

A large number of advances in the design of interactive search systems are attainable given recent technological developments, coupled with an expansion in the range of ways in which people employ search technology as it becomes more pervasive. In light of these developments, search technology will play an increasingly important role in how people tackle problems and challenges in the world. Beyond utilitarian goals, such as addressing information needs or solving problems, search is expanding into new important domains such as casual-leisure search (e.g., searching for fun; see Ross [1999], Fulton [2009], and Elsweiler et al. [2011]), diagnostic search (e.g., self-diagnosing a medical condition or debugging a problem with computer code; see Cartright et al. [2011]), and social searching (e.g., searching for people or information about people; see Evans and Chi [2008] and Horowitz and Kamvar [2010]). There are a number of new directions in the area of search interaction that will receive significant attention over the next decade and beyond. These advances can be framed in terms of technologies and opportunities. I touched on some of these earlier in the book, but in this chapter I provide more detail on each of these areas.

Mobile and Cloud Computing

The growth in mobile computing, in particular the ownership and use of smartphones and other mobile devices such as tablets and slates, has led to an increase in the volume of search traffic on new devices, especially in relevant search segments such as local search. Beyond query volume, however, the range of tasks that people are attempting on their devices will also grow to accompany an evolution in user expectations. There are challenges associated with the transition from desktop to mobile searching that need to be addressed. For example, given the limited real screen estate on mobile devices and the difficulties associated with typed query input, the best search experience is yet to be determined (and we currently fall back to trusted presentation methods). Given different interaction paradigms, richer models of interaction with mobile devices (e.g., Guo et al., 2013), as well the development of dedicated search applications for these devices, are both important in next-generation search systems.

A range of methodologies can be used to evaluate search systems both in the laboratory and in the real world. Each has advantages and disadvantages. Methodologies range from offline test collections and simulations of people's search behavior, to living laboratories, instrumented panels, ethnography, and large-scale log analysis. Specific objectives determine the metrics and methodologies that are employed.

There are at least three important dimensions of the available methods: (1) stage: the point in the design process that the method is used (i.e., formative, as the experimental system is being designed – with outcomes being able to inform design decisions, and summative, once design is complete and the system has been developed); (2) scale: the scale at which the method is employed (small, medium, or large); and (3) participants: the people involved, e.g., searcher simulations, user study subjects, internal users (company employees test their own products; a process known as dogfooding [Harrison, 2006]), or external customers (either via parallel flighting or interleaving of the output of alternative algorithms). Dumais et al. [2014] divided a subset of experimental methodologies into two variants: (1) observational (where people may be observed searching naturally) and (2) experimental (where the search experience may be intentionally manipulated per an experimental design) (see Table 11.1).

The various methods provide different perspectives on search system performance. Although the methods are listed separately in Table 11.1 and discussed separately in this chapter, combinations of different methods are valuable to help search providers develop a more complete understanding of system effectiveness.

The focus in search system evaluation has been on designing experiments that are: insightful and able to assess successfully the attributes on which they focus; affordable with respect to the cost of creating and running the experiments; repeatable so that others can build on results; and explainable so that they can guide subsequent improvements (Liu and Oard, 2006). The primary focus has been component evaluation, involving the isolation and control of experimental variables that can interfere with the reliability of conclusions drawn.

Current methods are insufficient for evaluating complex systems for a number of reasons (Kelly, 2009), including the inadequacy of user models and task models for capturing all types of information-seeking tasks, activities, and situations. Web and search engine indices are constantly changing.

As technology continues to evolve, many opportunities for advances in the development of search systems are emerging. Over the next decade and beyond, people will interact with search systems in new ways. There will be new, more natural and richer, interaction capabilities, as well access to knowledge mined from both data and human resources to help answer searchers’ questions. With the move beyond the current reactive model, there are significant and exciting challenges in how to make search systems assume the role of a companion or intelligent assistant, in terms both of understanding query semantics as a human intermediary would, and – given that understanding – search the world's knowledge to get people to answers and information quickly. This support will help searchers explore and learn about new topics and domains, and facilitate effective decision making and action.

Despite being a fairly recent phenomenon, microblogging has attracted a large number of researchers and practitioners who consider microposts a suitable source of data to ascertain public opinion. Among the reasons for that interest, we may find the fact that one single platform (i.e., Twitter) is the default choice for users; the ease with which one can collect data using public application programming interfaces (APIs); and the brevity of microposts, which forces users to get to the point when discussing any given topic.

This chapter is focused on efforts to exploit Twitter data to scrutinize public opinion in general, and political discussion in particular. It covers representative case studies conducted during the late 2000s and early 2010s and discusses their respective limitations. Finally, we analyze the implications of such approaches to political opinion in Twitter and depict important lines of research to further advance the field.

Introduction

This chapter is devoted to the problem of exploiting Twitter data to take the pulse of public opinion, particularly with regard to electoral forecasting. However, most of the arguments exposed here are not limited to Twitter, but apply broadly to any social networking site.

Twitter is (at the moment of this writing) the most convenient way to obtain user-generated content of opinionated nature about current events. That is the main reason why so much research has been performed on that platform and, in turn, such high expectations have been put on mining Twitter data.

For an in-depth overview of Twitter, in particular its historical evolution, consult the work by Van Dijck (2013, pp. 68–88). Rogers (2013) also analyzes Twitter's evolution, but mainly from a research perspective. His work shows the way in which research has evolved together with the service. In this regard, he provides compelling arguments to drop, once and for all, the caricatured image of Twitter as “pointless babble” and to acknowledge finally that it “serves as a mean to study cultural conditions.” Indeed, that simple idea pervades this chapter and the rest of the book.

It has been hypothesized that the language of Twitter users is associated with the socioeconomic well-being those users experience in their physical com- munities (e.g., satisfaction with life in their states of residence). To test the relationship between language use and psychological experience, researchers textually processed tweets to extract mainly sentiment and subject matters (topics) and associated those two quantities with census indicators of well-being. They did so by focusing on geographically coarse-grained communities, the finest-grained of which were U.S. census areas. After briefly introducing those studies and describing the common steps they generally take, we offer a case study taken from our own work on geographically smaller communities: London census areas.

Introduction

Happiness has often been indirectly characterized by readily quantifiable economic indicators such as gross domestic product (GDP). Yet in recent years, policy makers have tried to change that and introduced indicators that go beyond merely economic considerations. In 2010, the former French president Nicolas Sarkozy intended to include well-being in France's measurement of economic progress (Stratton, 2010). The UK prime minister David Cameron has been initiating a series of policies, under the rubric “Big Society,” that seek to make society stronger by getting more people running their own affairs locally all together. The idea shared by many governments all over the world is to explore new ways of measuring community well-being and, as such, put forward policies that promote more quality of life (happiness) rather than material welfare (GDP).

Measuring the well-being of single individuals can be successfully accomplished by administering questionnaires such as the Satisfaction with Life (SWL) test, whose score effectively reflects the extent to which a person feels that his or her life is worthwhile (Diener, Diener, & Diener, 1995). To go beyond single individuals and measure the well-being of communities, one could administer SWL tests to community residents. But that would be costly and is thus done on limited population samples and once per year at best.

Twitter is a new data channel for emergency managers to source public information for situational awareness and as a means of engaging with the community during disaster response and recovery activities. Twitter has been used successfully to identify emergency events, obtain crowd sourced information as the event unfolds, provide up-to-date information to the affected community from authoritative agencies, and conduct resource planning.

Introduction

Motivation

Natural disasters have increased in severity and frequency in recent years. According to Guha-Sapir et al. (2011), in 2010, 385 natural disasters killed over 297,000 people worldwide, impacted 217 million human lives, and cost the global economy an estimated US$123.9 billion. There are numerous examples from around the world: the 2004 Indian Ocean earthquake and tsunami; the more recent 2011 Tōhoku earthquake and tsunami, which damaged the Fukushima nuclear power station; hurricanes Katrina and Sandy in 2005 and 2012 respectively; the 2010 China floods, which caused widespread devastation; and Victoria's 2009 “Black Saturday” bushfires in Australia, killing 173 people and having an estimated A$2.9 billion in total losses (Stephenson, Handmer, & Haywood, 2012).

With urban development occurring on coastlines and spreading into rural areas, houses and supporting infrastructure are expanding into high-risk regions. The growing world population is moving into areas progressively more prone to natural disasters and unpredictable weather events. These events have been increasing in frequency and severity in recent years (Hawkins et al., 2012).

It has been recognized that information published by the general public on social media is relevant to emergency managers and that social media is a useful means of providing information to communities that may be impacted by emergency events (Lindsay, 2011; Anderson, 2012). To prepare and respond to such emergency situations effectively, it is critical that emergency managers have relevant and reliable information. For example, bushfire management is typically a regional government responsibility, and each jurisdiction has its own agency that takes the lead in coordinating community preparedness and responding to bushfires when they occur.

Twitter is a social network with over 250 million active users who collectively generate more than 500 million tweets each day. In social sciences research, Twitter has earned the focus of extensive research largely due to its openness in sharing its public data. Twitter exposes an extensive application programming interfaces (APIs) that can be used to collect a wealth of social data. In this chapter, we introduce these APIs and discuss how they can be used to conduct social sciences research. We also outline some issues that arise when using these APIs, and some strategies for collecting datasets that can give insight into a particular event.

Introduction

Twitter is a rich data source that provides several forms of information generated through the interaction of its users. These data can be harnessed to accomplish a variety of personalization and prediction tasks. Recently, Twitter data have been used to predict things as diverse as election results (Tumasjan et al., 2010; c.f. Chapter 2) or the location of earthquakes (Sakaki et al., 2010; c.f. Chapter 6). Twitter currently has over 250 million active users who collectively generate more than 500 million tweets each day. This creates a unique opportunity to conduct large-scale studies on user behavior. An important step before conducting such studies is the identification and collection of data relevant to the problem.

Twitter is an online social networking platform where the registered users can create connections and share messages with other users. Messaging on Twitter is unique, as messages are required to be at most 140 characters long, and these messages are normally broadcast to all the users on Twitter. Thus, the platform provides an avenue to share content with a large and diverse population with few resources. These interactions generate different kinds of information. Information is made accessible to the public via APIs or interfaces where requests for data can be submitted. In this chapter, we introduce different forms of Twitter data and illustrate the capabilities and restrictions imposed by the API on Twitter data analysis.

More than 500 million tweets are shared on Twitter each day, each devoting 140 characters to anything from announcements of dinner plans to calls for revolutions. Given its sheer size and the fact that most of its content is public, it is not surprising that Twitter has been extensively used to study human behavior on a global scale.

This book uses thematically grouped case studies to show how Twitter data can be used to analyze phenomena ranging from the number of people infected by the flu, to national elections, to tomorrow's stock prices. The idea for the book grew out of a three-hour tutorial on “Twitter and the Real World” given in October 2013 at the Conference on Information and Knowledge Management. Expanding on the topics covered in that tutorial, the book provides a wider thematic scope of research and the most recent scientific work.

All chapters are written by leading domain experts and take the reader to the forefront of the emerging new field of computational social science, using topical applications to illustrate the possibilities, advantages, and limitations of large, semi structured social media data to gain insights into human behavior and social interaction. Although most of the authors are computer scientists, the book is intended for readers who have not been exposed to formal training on how to analyze “big data.” To this end, unnecessary implementation details are avoided. Chapter 1, “Analyzing Twitter Data,” introduces readers to the tools and skills used in these studies and gives pointers on how to take the first steps as a novice researcher.

The opening chapter lays the groundwork for the book by surveying the opportunities and challenges of a “Twitter socioscope.” The chapters that follow are written to be stand-alone case studies that can be reordered according to the reader's needs and preferences.

Twitter will undoubtedly evolve and its usage will change, but even if it comes to be replaced by “the next big thing,” the detailed time-stamped records of hundreds of millions of global users will continue to be one of the most valuable sources of data on human behavior ever assembled, and these case studies will remain useful as an introduction to the methods, research opportunities, and challenges that these data present.