13 - Evaluating Deployed Decision-Support Systems for Security: Challenges, Analysis, and Approaches

Summary

Introduction

As discussed in other chapters of this book, an increasing number of technically sophisticated tools are available to support decisionmaking for security resource allocation in many different domains. In this chapter we discuss the question of evaluating these deployed security systems, using examples from our own research to illustrate some of the key challenges in doing evaluation for security systems. Two of the primary difficulties are (1) that we cannot rely on adversaries to cooperate in evaluation, which makes it difficult to validate models, and (2) there is (thankfully) very little data available about real-world terrorist attacks.

Despite the difficulties of comprehensive evaluation in security domains, it is only by asking the question, how well does a system work? that policy makers can decide how to allocate finite resources to to different security measures. We discuss the goals of security systems, the elements that comprise these systems, and different approaches for evaluation. Every approach has drawbacks, so in lieu of an ideal test, we advocate a comprehensive style of evaluation that uses diverse metrics and data to perform cost-benefit analysis for the complete system. We also emphasize that the focus of the evaluation is not, is system X the perfect security system? which is an impossible standard. Rather, the relevant question is which of the available alternatives should be used? Providing strong evidence that one alternative is superior to other approaches is often feasible, even when providing exact quantitative measures of value is not.