Overview of the Organization

The British Library (BL) is the national library of the United Kingdom. Its work is supported by the expertise of more than 2,000 staff and by links with other organizations worldwide. The library provides:

• Internationally important reading room and inquiry services

• The world's leading document supply services

• Specialist information services in key subject areas

• On-line and Web-based services

• Essential services for the library, archives, and information world

• Library facilities for the general public.

The library's outstanding collection, developed for more than 250 years, contains more than 150 million items and represents every age of written civilization, every written language, and every aspect of human thought.

BL can be seen as the most complete of all the case studies, embodying aspects of each of the process elements described in Part 1 of this book.

Characteristics of the Testing Requirement

BL has a significant reliance on software systems to support its business for the purposes of:

• Maintaining very large quantities of information on document holdings

• Maintaining extensive lender personal details

• Recording details of document borrowings

• Maintaining on-line catalogue searching (involving millions of records)

• Supporting large-scale document supply.

The majority of these systems need to interoperate in order to inspect or exchange data and/or to invoke operations on other systems. The systems in use are of various ages, based on a variety of implementation technologies, and frequently supported on different operating systems and hardware platforms.

Introduction

Irrespective of the technology employed in the development of a software system, any testing process will involve a number of common activities that must be performed by staff with appropriate skills and completed in a particular order to ensure successful testing – this is the essence of a repeatable, predictable, reusable process.

Where an organization has a requirement to test a number of different applications, each of which may have been developed using different tools, techniques, and languages, rather than emphasizing the differences between the applications and their implementation technologies, it is much more beneficial to look for the similarities.

Where differences that impact on the testing process are observed, these exceptions should be documented, and the testing solutions that are adopted to address them incorporated into the process as supplementary activities for future use.

Thus the starting point for determining how to approach the testing of any new or novel technology should be to assume that the testing will fit into the currently used process, and not throw current best practice away in favor of creating a new process from first principles.

This appendix discusses the particular issues involved in testing object-oriented (OO) software systems (36) and makes a number of proposals for how to approach this activity. Specifically, Section R2 briefly reviews the process of object-oriented software development, Section R3 discusses the impact of OO on the management and planning of testing, and Section R4 examines the impact of OO on the design of tests.

Introduction

This appendix contains guidance on ensuring that the reuse of tests and testing materials is achieved between testing phases and testing projects. It includes:

• How and when to create a reuse pack

• A list of the typical contents of a reuse pack

• Guidance on using reuse packs.

Creating a Reuse Pack

At the end of any testing phase, the test team leader (or development team leader during unit and integration testing) should collect copies of a number of items generated during the testing project into a single file (a simple A4 manila folder is usually sufficient for this purpose). This file is termed a reuse pack.

The reuse pack will be archived with the rest of the project files and can be subsequently retrieved on request from the testing manager (or equivalent role).

The contents of the reuse pack can be used by other testing projects to reduce the effort and cost associated with testing and to improve the consistency and quality of testing.

Examples of the use of reuse packs include:

• During subsequent testing phases of the same application under test (AUT) (such as reusing some system tests during acceptance testing)

• In regression testing modifications to the same AUT (such as either rerunning a complete system test or selected elements of the system test)

• Where the AUT is similar or related to another AUT (such as employing existing integration test material when an additional module is added to an existing suite of applications).

Introduction

This chapter deals with the issues associated with the organization, management, and high-level planning of the testing process.

A generic approach to organizing the testing process is described, which can be applied to businesses involved in developing and/or testing software. The chapter discusses the implications for the testing process of the size of the business, its prevailing quality culture, and (where appropriate) the maturity of its software development process.

The case studies in Part 2 of this book provide specific examples of how a number of different businesses have customized this generic approach in response to their own particular organizational, management, and testing requirements.

This chapter also considers a number of key requirements that must be considered in implementing a rigorous, effective, and efficient testing process: the need for configuration management, defect tracking, and the need to ensure that each of the requirements of the application under test (AUT) has been verified. The issues associated with each of these requirements are discussed, and recommendations for supporting them are provided.

The chapter concludes with a discussion of the role of risk-based testing and its use in the management and planning of testing projects.

Overview of the Organization

More than 521,000 users in 52,800 locations access Reuters information and news worldwide. Data is provided on more than 940,000 shares, bonds, and other financial instruments as well as on 40,000 companies. Financial information is obtained from 260 exchanges and over-the-counter markets, and contributed by 5,000 clients.

Reuters services are delivered globally over one of the world's most extensive private satellite and cable communications networks. Typically, Reuters updates 6,000 prices and items of other data per second and handles 65 million changes daily. The information available from Reuters' databases ranges from real-time to greater than 10 years old.

The company's two main business divisions are Reuters Information (RI) and Reuters Trading Systems.

Reuters Information products include real-time information and historical information databases, and focus on four main markets, foreign exchange and money, commodities (including energy), fixed income, and equities. Reuters corporate and media information business includes textual news services for print media, broadcast, and on-line clients. The company prides itself on the diverse content, freedom from bias, accuracy, and speed of supply of information to its customers.

Reuters Trading Systems division groups together Reuters management systems, transaction products, risk management products, and other applications.

The Reuters Product Acceptance Group (PAG) operates within the RI division and is based in Singer Street, London. PAG is responsible for the acceptance testing of financial trading products on behalf of its customers, which are implemented under contract by third-party suppliers.

Introduction

This appendix provides recommendations for the terms of reference (TORs) for the following staff involved in software testing:

• Testing manager

• Test team leader

• Test analyst

• Tester

• Independent test observer.

For each TOR, the following information is documented:

• Responsibilities

• Reporting and liaison

• Typical characteristics

• Appropriate references for further information.

Each TOR appears in the following pages of this appendix.

This appendix also describes the role of a typical testing programme board (see Chapter 4) for those organizations interested in introducing such an entity into their testing programme.

Individual TORs can be provided to the appropriate staff on testing projects, either as photocopies or electronically, as memory aids.

Introduction

It is often the case that test practitioners have been tasked with delivering quality improvements in their projects or organization, but do not have the luxury of sufficient time, resources, or budget to devise, design, implement, and roll out a fully formed and working testing process. This appendix provides a real-world example describing how it is still possible to adopt a subset of the best practices described in this book to gain immediate, “quick win,” quality improvements.

Test Process Adoption – a Real-World Example

Introduction

As a testing practitioner who has been tasked with improving the quality of the software delivered by your project or by your organization, where do you start the process of adopting and using quality management best practices?

What you have read so far may appear extremely virtuous, but if you do not have sufficient time, resource, or budget, it may appear to be too much change to bring about in too short a time. Grabbing a few users, with no plan and no goal, however busy everyone appears to be, is no substitute for deliberate acts of testing.

If you are able to recognize your predicament as being near the bottom of a waterfall development (69), with a delivery that is in danger of missing the deadline, your testing resource appears scant and the implementation date cannot be moved, then you are not alone!

Introduction

This appendix contains a test script template that may be copied and used to record the information necessary for conducting each test comprising a particular testing phase. An electronic version of the template can be obtained using the following link: http://www.cambridge.org/9780521148016/.

A completed example of a test script (Figure E1) is also included at the end of this appendix for guidance.

For each test, a separate test script will need to be completed by the test analyst. Each test script contains the following sections:

• Project ID – the unique project identifier

• AUT title – the definitive title of the application under test (front sheet only)

• AUT version – the definitive version information for the AUT (front sheet only)

• Testing phase – the title of the phase of testing being conducted (e.g., unit test, integration test, system test, etc.) (front sheet only)

• Date of test – the planned start date of testing (front sheet only)

• Test ID – the unique identifier for the test

• Purpose of test – a brief description of the purpose of the test including a reference where appropriate to the requirement that is to be tested (consider providing references to the requirements specification, design specification, user guide, operations guide, and/or installation guide), as well as any dependencies from or to other test scripts/test cases

• […]

Introduction

This appendix contains a number of one-page testing guides that can be used as an aide memoir for staff involved in the various phases of software testing. Full details of each of the testing phases covered in this appendix are provided in Chapters 5 to 11.

In particular, the guides are intended to be used by the tester (whether they be a member of the development team (in unit and integration testing), a member of the testing team or the quality assurance group.

This appendix contains the following testing guides:

• Unit testing guide

• Integration testing guide

• System testing guide

• Systems integration testing guide

• User/Operations acceptance testing guide

• Regression testing guide.

For each testing phase, the following summary information is provided:

• Purpose – a brief description of the purpose of this testing phase

• Approach – listing the high-level approach to conducting this testing phase

• Inputs – listing the inputs to this testing phase

• Testing techniques – listing the testing techniques appropriate to this testing phase

• Outputs – listing the outputs from this testing phase

• Quality considerations – listing a number of quality issues that should be considered within this testing phase

• References – further sources of information on this testing phase.

The individual testing guides should be provided to appropriate staff on testing projects, either as photocopies or electronically for reference purposes.

Introduction

This appendix contains an example illustrating the testing technique of pairwise testing described in Chapter 3 that the test analyst can use to select specimen data for use in testing the application under test (AUT).

The Testing Problem

Continuing with the hotel room booking theme, when the employee comes to book a room using an on-line booking system on the hotel Web site, he or she may have to make a selection using radio buttons of whether he or she wants a single, double, or family room. The employee may also have to decide whether he or she would like a bath or a shower in the room and whether he or she needs smoking or nonsmoking accommodation.

Introduction

This chapter discusses the “recent” phenomenon of agile approaches to software development and testing, reviews a number of successful agile quality management practices being employed by testing practitioners on real-world projects, and concludes by making a series of recommendations about how to implement an effective and efficient agile testing approach.

Chapter 20 provides a case study that documents the role and use of agile testing best practices within the context of an agile development and testing project.

Overview of Agile Testing

In recent years new approaches to the development and testing of software and the management of software projects, such as Extreme Programming (XP) and Scrum (50 and 51) have encouraged practitioners to be more agile in the delivery of software systems.

The typical features of such agile approaches include:

• Breaking large monolithic software projects down into smaller more easily achieved iterations

• The rapid delivery of working elements of the developing system for early testing and to gain feedback from the customer on the suitability of the software

• Colocation of the development and testing team with suitably knowledgeable and empowered customer representatives

• Employing use cases (8) or user stories (50) to capture customer requirements in a clear and simple manner that enables the customer to understand and agree on the requirements

• Being as responsive to customer requests for changes to the developing system as possible

• Employing focused and effective project communications including stand-up meetings (51), agile retrospectives (53), and short lines of communications

• […]

Introduction

Even the very best software testing process is nothing without the acceptance and consistent use by a community of testers within an organization. Technical issues aside, the introduction and adoption of a testing process is likely to be the most difficult step in providing a formal, rigorous, and reusable process within an organization.

This chapter examines the issues involved in introducing a testing process into an organization, and the management of its successful adoption and use. This chapter also addresses the issues associated with the subsequent maintenance and update of the testing process, and its ongoing evolution and enhancement.

The chapter concludes by providing process adoption guidance for those test practitioners who may be under pressure to deliver immediate quality improvements, but who may not have the luxury of sufficient time, resources, or budget.

Introduction and Adoption of a Testing Process

Overview

This section discusses the need to first establish that there is a requirement for a formal testing process within an organization, and where this is the case, goes on to discuss the need to develop a strategy for the introduction and adoption of the process.

Overview of the Organization

Confirmit supplies software that enables organizations to conduct customer feedback, employee feedback, and market research programs.

The Confirmit customer base is drawn primarily from global 5000 companies and worldwide market research agencies Its customers rely on Confirmit's wide range of products for feedback/data collection, panel management, data processing, analysis, and reporting.

Confirmit takes pride in providing innovative, superior technological solutions that are scalable, secure, and flexible to support its clients' constantly evolving needs. The company is committed to understanding and meeting the needs of its multinational customer base and developing strategic relationships that support this. Confirmit's adoption and use of Tom Gilb's evolutionary project management (67) approach to agile product development and testing is a key factor in achieving these goals.

Confirmit was established in 1996 and rapidly became the leading provider of software-as-a-service (68) feedback and research solutions. In 2007, Confirmit merged with leading market research software provider Pulse Train. Since the merger, Confirmit has evolved its product portfolio to offer a range of best-of-breed solutions based on the combined forty years of software expertise and market knowledge of the two companies. In 2009, this evolution led to the launch of Confirmit Horizons, the world's first fully on-demand, multimode platform for market research, customer feedback, and employee feedback.

Confirmit now employs more than 260 people and has offices in Oslo, Norway (headquarters); Guildford and London in the United Kingdom; New York and San Francisco in the United States; and Moscow and Yaroslavl in Russia.

Why is astronomy considered a science while astrology is considered only a pseudoscience? In other words, how can we prove that a theory faithfully describes reality, and that this theory can then be used to predict unknown facts? Karl Popper, the well-known philosopher, studied these problems and summarized his conclusions in one phrase: “The criterion of the scientific status of a theory is its falsability, or refutability, or testability.” For Popper, “confirming evidence should not count except when it is the result of a genuine test of the theory.”

The testing process of a scientific theory is quite similar to the process of providing confirmation either to risky predictions or to attempts to falsify that theory. Testing is a complex activity. It has to simultaneously bear in mind the theory and the external reality; it has to provide objective answers to complex questions related to our own perceptions of a rational reality.

When developing software, we follow the same thought process, since one builds an abstract model between the external world and the user. In our software, we define strict processes that will guide our actions, and we build the data we want to manipulate in complex databases and templates.

Overview

The objective of integration testing is to demonstrate that modules comprising the application under test (AUT) interface and interact together in a correct, stable, and coherent manner prior to system testing. Integration testing is typically conducted by the development team and involves independent observation of the testing process. Integration testing may also be termed link or module testing, but it should not be confused with systems integration testing (see Chapter 8).

Testing should be performed against the functional requirements of the AUT using black box testing techniques. Test case design should demonstrate correct interfacing and interaction between modules but should avoid duplication of unit testing effort (however, where appropriate, reuse of unit tests should be considered to reduce testing timescales and effort).

The precise definition of a “module” depends on the approach to design of the AUT and the implementation technology. For example:

• A module in an application developed using a procedural programming language could be represented by a closely related group of functions or procedures that perform a well-defined service within the AUT and that communicate with other component modules via strictly defined interfaces

• A module in an application developed using an object-oriented programming language could be represented by a collection of objects that perform a well-defined service within the AUT and that communicate with other component modules via strictly defined interfaces

• […]

Introduction

This appendix contains a list of criteria to be used in evaluating automated software testing tools and a suggested approach to conducting the evaluation. A simple scoring scheme with weightings is described, which can be used to score each of the evaluation criteria. The values for weightings and criteria scores may be modified through experience and use.

An evaluation summary checklist can be found at the end of this appendix, which can be copied and used in evaluating testing tools.

An electronic version of this appendix can be obtained using the following link: http://www.cambridge.org/9780521148016 so that the evaluation criteria and template can be customized to more closely match the evaluation requirements of the reader.