This chapter uses logical expressions to define criteria and design tests. This continues our progression into the RIPR model by ensuring that tests not only reach certain locations, but the internal state is infected by trying multiple combinations of truth assignments to the expressions. While logic coverage criteria have been known for a long time, their use has been steadily growing in recent years. One cause for their use in practice has been standards such as used by the US Federal Aviation Administration (FAA) for safety critical avionics software in commercial aircraft.

As in Chapter 7, we start with a sound theoretical foundation for logic predicates and clauses with the goal of making the subsequent testing criteria simpler. As before, we take a generic view of the structures and criteria, then discuss how logic expressions can be derived from various software artifacts, including code, specifications, and finite state machines.

This chapter presents two complementary approaches to logic testing. The first, whichwe call semantic logic coverage, considers what logic expressions mean regardless of how they are formulated. The strength of the semantic approach is that we get the same tests even if the predicate is rewritten in a different but equivalent form. The semantic approach is more common and more likely to be familiar to readers. The second approach, which we call syntactic logic coverage, develops tests specifically tailored to how a logic expression is formulated. The strength of the syntactic approach to logic coverage is that it addresses the specific ways in which an engineer might incorrectly formulate a given logic expression.

Studies have found that the syntactic approach usually detects more faults, but the test criteria are relatively complicated and can be quite expensive. In recent years, the research community has found ways to reduce the number of tests required without sacrificing fault detection. Specifically, the number of tests required for syntactic coverage has dropped substantially–to the point where it is competitive with the semantic approach. While the safety community still relies on the semantic approach, it may be time for this community to consider the syntactic approach.

Regression testing is the process of re-testing software that has been modified. Regression testing constitutes the vast majority of testing effort in commercial software development and is an essential part of any viable software development process. Large components or systems tend to have large regression test sets. Even though many developers do not want to believe it, (even when faced with indisputable evidence!), small changes to one part of a system often cause problems in distant parts of the system. Regression testing is used to find this kind of problem. In practice, regression testing is often incorporated into a continuous integration service, as discussed in Chapter 4.

It is worth emphasizing that regression tests must be automated. Indeed, it could be said that unautomated regression testing is equivalent to no regression testing. A wide variety of commercially available tools are available. The current most common are JUnit and its derivative cousins (HTMLUnit, Selenium, SUnit, CUnit, TestNG, NUnit, CPPUnit, PHPUnit, etc.).

Capture/replay tools automate testing of programs that use graphical user interfaces. Version control software, already in use to manage different versions of a given system, effectively manages the test sets associated with each version. Scripting software manages the process of obtaining test inputs, executing the software, marshaling the outputs, comparing the actual and expected outputs, and generating test reports.

The aim of this section is to explain what kinds of tests should be in a regression test set, which regression tests to run, and how to respond to regression tests that fail. We treat each of these issues in turn. We direct the reader interested in detail on any of these topics to the bibliographic notes.

The test engineer faces a Goldilocks problem1 in determining which tests to include in a regression test set. Including every test set possible results in a regression test set that is too large to manage. The result is that the test set cannot be run as often as changes are made to the software. For traditional development processes, this period often amounts to a day; regression tests run at night to evaluate software changed that day, with developers reviewing the results the following morning. For agile development processes the period is significantly shorter, for reasons discussed in Chapter 4.

The purpose of this book is to teach software engineers how to test. This knowledge is useful whether you are a programmer who needs to unit test your own software, a full-time tester who works mostly from requirements at the user level, a manager in charge of testing or development, or any position in between. As the software industry moves into the second decade of the 21st century, software quality is increasingly becoming essential to all businesses and knowledge of software testing is becoming necessary for all software engineers.

Today, software defines behaviors that our civilization depends on in systems such as network routers, financial calculation engines, switching networks, theWeb, power grids, transportation systems, and essential communications, command, and control services. Over the past two decades, the software industry has become much bigger, is more competitive, and has more users. Software is an essential component of exotic embedded applications such as airplanes, spaceships, and air traffic control systems, as well as mundane appliances such as watches, ovens, cars, DVD players, garage door openers, mobile phones, and remote controllers. Modern households have hundreds of processors, and new cars have over a thousand; all of them running software that optimistic consumers assume will never fail! Although many factors affect the engineering of reliable software, including, of course, careful design and sound process management, testing is the primary way industry evaluates software during development. The recent growth in agile processes puts increased pressure on testing; unit testing is emphasized heavily and test-driven development makes tests key to functional requirements. It is clear that industry is deep into a revolution in what testing means to the success of software products.

Fortunately, a few basic software testing concepts can be used to design tests for a large variety of software applications. A goal of this book is to present these concepts in such a way that students and practicing engineers can easily apply them to any software testing situation.

This textbook differs from other software testing books in several respects. The most important difference is in how it views testing techniques. In his landmark book Software Testing Techniques, Beizer wrote that testing is simple—all a tester needs to do is “find a graph and cover it.”

Like other software, tests can be designed in an abstract way. However, as discussed at length in Chapter 4, developers want tests to become “real” as soon as possible so that they can benefit from the immediate feedback of any failed tests. To do so, all the code must compile, tests must not cause collateral damage, the process must be repeatable, and it must complete in a timely manner. Unit testing normally doesn't pose a serious challenge with respect to these constraints, but other test phases certainly do. This section discusses technical problems that arise during the implementation of test cases. We do not catalog the problems from a process point of view. Instead, we focus on the technical strategies used to solve these problems. Many of the problems arise naturally during software integration, but integration is not the only source of difficulty; the testing of fully integrated systems also requires the techniques discussed here.

Software programs are composed of many pieces of software of varying sizes. Individual programmers are often responsible for testing the lowest level components (classes, modules, and methods). After that, testing must be carried out in collaboration with software integration. Software can be integrated in many ways.

Integration testing is the testing of incompatibilities and interfaces between otherwise correctly working components. That is, it is the testing needed to assure correct integration of subcomponents into a bigger working component. This is emphatically not the same as testing an already integrated component.

Integration testing is often done with an incomplete system. The tester may be evaluating how only two of many components in the system work together, may be testing integration aspects before the full system is complete, or may be putting the system together piece by piece and evaluating how each new component fits with the previously integrated components.

This chapter uses the term software “component” in a very broad sense: A component is a piece of a program that can be tested independently of the complete program or system. Thus, classes, modules, methods, packages, and even code fragments can be considered to be components. Also, non-executable software artifacts such as XML files, XML schemas, and databases can be considered to be components.

This chapter introduces one of the major innovations in the second edition of Introduction to Software Testing. Software testing is inherently complicated and our ultimate goal, completely correct software, is unreachable. The reasons are formal (as discussed below in section 2.1) and philosophical. As discussed in Chapter 1, it's not even clear that the term “correctness” means anything when applied to a piece of engineering as complicated as a large computer program. Do we expect correctness out of a building? A car? A transportation system? Intuitively, we know that all large physical engineering systems have problems, and moreover, there is no way to say what correct means. This is even more true for software, which can quickly get orders of magnitude more complicated than physical structures such as office buildings or airplanes.

Instead of looking for “correctness,” wise software engineers try to evaluate software's “behavior” to decide if the behavior is acceptable within consideration of a large number of factors including (but not limited to) reliability, safety, maintainability, security, and efficiency. Obviously this is more complex than the naive desire to show the software is correct.

So what do software engineers do in the face of such overwhelming complexity? The same thing that physical engineers do–we use mathematics to “raise our level of abstraction.” The Model-Driven Test Design (MDTD) process breaks testing into a series of small tasks that simplify test generation. Then test designers isolate their task, and work at a higher level of abstraction by using mathematical engineering structures to design test values independently of the details of software or design artifacts, test automation, and test execution.

A key intellectual step in MDTD is test case design. Test case design can be the primary determining factor in whether tests successfully find failures in software. Tests can be designed with a “human-based” approach, where a test engineer uses domain knowledge of the software's purpose and his or her experience to design tests that will be effective at finding faults. Alternatively, tests can be designed to satisfy well-defined engineering goals such as coverage criteria. This chapter describes the task activities and then introduces criteria-based test design.

Part I of this book laid down the foundations for modern software testing, and part II went into great detail about technical methods to design effective test case values from criteria. Eventually, of course, the concepts must be put into practice. This brings in many additional pragmatic concerns. Part III of this book provides a summary overview of the major aspects of putting the Model-Driven Test Design process into practice. The most obvious audience for these chapters are test team managers. We start with overall process concerns in this chapter, then discuss aspects of practical testing such as test plans, integration testing, regression testing, and the design and implementation of test oracles.

OVERVIEW

Many organizations postpone all software testing activities to the end of development, after the implementation has started, or even after implementation has ended. By waiting until late in the process, testing ends up compromised. Not enough resources (time or budget) remain, problems with previous stages have been solved by taking time and dollars from testing, and we do not have enough time to plan for testing. Instead of planning and designing tests, the developers have time only to run tests, usually in an ad hoc manner. The key point is that the goal is to create high-quality software, and the old adage that “quality cannot be tested in” is still very relevant. A tester cannot show up at the last minute and make a bad product good; high quality has to be part of the process from the beginning.

This section discusses how to integrate testing with development, where testing activities begin as soon as development activities begin, and are carried out in parallel with the development stages. Specific activities, including planning, active testing, and development-influencing activities, can be associated with each of the traditional lifecycle phases. These activities can be carried out by the developers or by separate test engineers, and can be associated with development stages within the confines of any specific development process. These testing activities allow the tester to detect and prevent faults throughout the software development process.

The role of testing in software development has undergone radical changes in recent years. Testing has evolved from afterthought to a central activity in certain development methods—particularly agile methods. This chapter explains the evolving role of testing in software development and highlights the key theoretical and practical enablers for that evolution. The message of this chapter is as follows: If high-quality testing is not centrally and deeply embedded in your development process, your project is at high risk for failure. Your project might fail in the technical sense, in that you simply lose control of what the code actually does. Or it might fail in the business sense, in that your competitors roll out better functionality faster. It does not really matter which way you fail; sadly, the consequences are the same.

TAMING THE COST-OF-CHANGE CURVE

Traditional software engineering, as described in standard texts on software engineering, came into being as a field precisely because the development of large software projects was proving to be increasingly difficult—even impossible—using ad hoc development methods. Traditional software engineering focuses primarily on extensive modeling and upfront analysis. The goal is to reveal potential problems and changes as early as possible. The economic rationale is that effort spent on revealing failures early delivers an enormous return on investment. Every software engineering text shows the traditional “cost-of-change” curve, where the key variable is the lag between when a change should ideally be made and when the need for that change is recognized. Figure 1.1 in Chapter 1 illustrates this concept, by showing that the cost of finding and fixing faults balloons as we move from unit testing to integration testing to system testing to deployment.

A more general description is shown in Figure 4.1. The way to read this figure is to identify the time at which a decision (or the mistake) is originally made and the time at which that decision is revised (or mistake is repaired). The time interval between these events is shown on the horizontal axis. Cost, shown on the vertical axis, is a function of the length of time between the original and the revision. The “delta” cost starts at a small value for revisions made shortly after the original and climbs ever more steeply as the interval between the two events grows longer.

One of the most widespread changes in software testing during the last decade has been the increased use of test automation. We introduced test automation in Chapter 2 as implementing tests into executable test scripts. This chapter expands on that concept, starting with a complete definition.

Software testing can be expensive and labor intensive, so an important goal of software testing is to automate as much as possible. Test automation not only reduces the cost of testing, it also reduces human error and makes regression testing easier by allowing a test to be run repeatedly with the push of a button.

Software engineers sometimes distinguish revenue tasks, which contribute directly to the solution of a problem, from excise tasks, which do not. For example, compiling a Java class is a classic excise task because, although necessary for the class to be executable, compilation contributes nothing to the behavior of that class. In contrast, determining which methods are appropriate to define a data abstraction in a Java class is a revenue task. Excise tasks are candidates for automation; revenue tasks usually are not. Software testing probably has more excise tasks than any other aspect of software development. Maintaining test scripts, rerunning tests, and comparing expected results with actual results are all common excise tasks that routinely use large amounts of test engineers’ time. Automating excise tasks serves the test engineer in many ways. First, eliminating excise tasks eliminates drudgery, thereby making the test engineer's job more satisfying. Second, automation frees up time to focus on the fun and challenging parts of testing, such as test design, a revenue task. Third, automation allows the same test to be run thousands of times without extra effort in environments where tests are run daily or even hourly. Fourth, automation can help eliminate errors of omission, such as failing to update all the relevant files with the new set of expected results. Fifth, automation eliminates some of the variance in test quality caused by differences in individual's abilities.

Previous chapters introduced coverage criteria and gave some simple examples. Now we are ready to define this important concept formally. This chapter presents the ideas behind criteria in an abstract way, applicable to all structures. The four chapters in Part II instantiate these ideas with specific criteria or specific structures and show how they are used in practice.

COVERAGE CRITERIA DEFINED

It is common to hear testers talk about “complete testing,” “exhaustive testing,” and “full coverage.” These terms are poorly defined because of a fundamental theoretical limitation of software. Specifically, the number of potential inputs for most programs is so large as to be effectively infinite. Consider a Java compiler—the number of potential inputs to the compiler is not just all Java programs, or even all almost-correct Java programs, but all strings. The only limitation is the size of the file that can be read by the parser. Therefore, the number of inputs is effectively infinite and cannot be explicitly enumerated.

This is where formal coverage criteria come in. Since we cannot test with all inputs, coverage criteria are used to decide which test inputs to use. The rationale behind coverage criteria is that they divide up the input space to maximize the number of faults found per test case. From a practical perspective, coverage criteria also provide useful rules for when to stop testing.

This book defines coverage criteria in terms of test requirements as introduced in Chapter 2. The basic idea is that we want our tests to have certain properties, each of which is provided (or not) by at least one test case.

Definition 5.23 Test Requirement: A test requirement is a specific element of a software artifact that a test case must satisfy or cover.

This definition is fairly abstract, and more specific versions for individual structures and criteria will be given in later chapters. Test requirements usually come in sets, and we use the abbreviation TR to denote a set of test requirements.

Test requirements can be described with respect to a variety of software artifacts, including the source code, design components, specification modeling elements, or even descriptions of the input space. Later in this book, test requirements will be generated from all of these.

This chapter introduces some of the most widely known test coverage criteria. This chapter uses graphs to define criteria and design tests. This starts our progression into the RIPR model by ensuring that tests “reach” certain locations in a graph model of the artifact being tested. The chapter starts with basic theory as a way to make the practical and applied portions of the chapter easier to follow. We first emphasize a generic view of a graph without regard to the graph's source. After this model is established, the rest of the chapter turns to practical applications by demonstrating how graphs can be obtained from various software artifacts and how the generic versions of the criteria are adapted to those graphs.

Directed graphs form the foundation for many coverage criteria. They come from many sources and types of software artifacts, including control flow graphs from source, design structures, finite state machines, statecharts, and use cases, among others. We use the term artifact in the most general way, to be anything associated with the software, including the requirements, design documents, implementation, tests, user manuals, and many others. Graph criteria usually require the tester to “cover” the graph in some way, usually by traversing specific portions of the graph. This overview presents graphs in general terms, and overlaps standard texts on discrete math, algorithms, and graph theory. Unlike those theoretical treatments, we focus only on the ideas needed for testing and introduce some new terminology that enable test design.

Given an artifact under test, the idea is to extract a graph from that artifact. For example, the most common graph abstraction for source code maps executable statements and branches to a control flow graph. It is important to recognize that the graph is not the same as the artifact, and usually omits certain details. It is also possible for the same artifact to have several useful, but different, graph abstractions. The same abstraction that produces the graph from the artifact also maps test cases for the artifact to paths in the graph. Accordingly, a graph-based coverage criterion evaluates a test set for an artifact in terms of how the paths corresponding to the test cases “cover” the artifact's graph abstraction.