Overview of the Organization

The Wine Society is a co-operative owned by its shareholder (its “members”). It was founded in 1874 to serve its members rather than be run for profit. This ethos still holds true today. An elected body of members oversees the Wine Society and represents the interests of more than 85,000 active members.

The objective of the society is to offer its members wines and spirits of good quality from a wide range of sources, at the best prices possible, which it has consistently managed to achieve, with a list of more than 800 different wines available either by the bottle, in unmixed dozens, or in mixed cases of the members' own selection. Preselected mixed cases are also available and come with comprehensive tasting notes. The Wine Society organizes delivery of member purchases to any address in the United Kingdom.

The Wine Society has extensive storage facilities with temperature-controlled warehouses in Stevenage. The longer-lived wines are allowed to mature gently and reach their optimum potential. It is also possible for members to take advantage of the society storage facilities by storing wines bought from the society in the designated members' reserve section of the cellars. Updated drinking dates are provided to the members taking advantage of this facility annually to appraise them of the state of maturity of their stock.

Introduction

This appendix contains a specimen test certificate template, which may be customized and used to formally record the successful completion of a specific test phase for a particular application under test (AUT). An electronic version of the template can be obtained using the following link: http://www.cambridge.org/9780521148016.

Although it is unlikely that a test certificate will be used to formally record the successful completion of the earlier phases of the testing process, in any formal acceptance of an AUT, and in particular from a third-party developer or supplier, the use of such a certificate should be mandatory.

When customizing the specimen test certificate template for a particular AUT under a specific testing phase, the following information should be considered for inclusion within the test certificate:

• The title of the AUT

• The test phase name

• A statement of acceptance describing the circumstances of the test, its date (and optionally, its location), and what the AUT has been tested against

• The signature of the member of staff accepting the tested AUT (must be of an appropriate level of seniority, particularly for acceptance of third-party bespoke developed systems or products). The signature should be dated

• The signature of the independent test observer (this item may not be appropriate for an acceptance test where the system has been supplied by a third-party organization). The signature should be dated

• The signature of the supplier or developer (these need not be the representatives of a third-party organization – for example, the AUT may have been developed in-house, with a suitably senior representative of the development project signing). The signature should be dated.

Introduction

This appendix contains a template for a test log, which may be copied and used to record the activities and events comprising a particular test phase. An electronic version of the template can be obtained using the following link: http://www.cambridge.org/9780521148016.

The purpose of a test log is to document the chronological record of all relevant details of a testing project (in effect, a test log is a diary of all relevant events that have occurred during the testing project).

The test log forms part of the formal documentation associated with the testing project and will be filed at the completion of the testing project.

The test log template presented on the following pages is based on that found in (1) with additional information from (15), plus extensive feedback obtained in implementing testing frameworks for a number of clients (see the case study section of this book).

Two different test log pages are presented on successive pages:

• Test log front sheet

• Test log continuation sheet.

Introduction

This appendix contains a template for a test plan document, guidance on its use, and where appropriate, examples of the typical contents of such a document. This template may be used as the basis of a test plan document for a particular test phase.

An electronic version of this template can be downloaded from the following link: http://www.cambridge.org/9780521148016 (or, assuming the role has been implemented, a copy can be obtained from the testing manager or equivalent role).

As described in Chapter 4 (The Management and Planning of Testing), each testing project must generate a test plan as one of the documents produced prior to the commencement of testing activities.

Where text appears within the body of the template in angle brackets (< >), this denotes a place marker, which must be replaced with the appropriate information for a particular testing phase.

Text that appears in italic is provided for information only (such as illustrative examples) and should not appear in the final test plan document.

Where references to “Appendices” appear within this appendix, these refer to other appendices in this book. The term “Annex” is used to refer to supplementary sections within this template.

The principal approach to be used in writing a test plan document is to keep it brief and simple. Make best use of the supporting information presented in this book (such as other relevant template appendices), as well as pertinent material from the appropriate testing phase chapters.

Overview

The objective of system testing is to establish confidence that the application under test (AUT) will be accepted by its users (and/or operators), that is, that it will pass its acceptance tests. During system testing, the functional and structural stability of the system will be demonstrated, as well as nonfunctional requirements such as performance and reliability. System testing is conducted by the testing team under the supervision of the test team leader.

Where the AUT has a moderate requirement to interoperate with one or more collaborating software systems, this may be tested during systems testing. Where the AUT has a significant requirement to interoperate with several collaborating software systems and/or the nature of the interoperation is complex, this will typically be tested during a separate testing phase – termed systems integration testing (see Chapter 8 for further details).

System testing should employ black box testing techniques and will test high-level requirements of the system without considering the implementation details of component modules (e.g., performing thread testing and/or testing complete transactions).

In terms of the V Model, system testing corresponds to the specification phase of the software development lifecycle (see Figure 7.1).

Testing issues associated with system testing (such as test planning and resourcing and the review of testing requirements and high-level test design), should be considered during the specification phase of system development.

Introduction

The idea that we should try to learn by our experiences is both natural and intuitive, but one we frequently seem to ignore in the software development and test context.

One particular approach to learning by experience in a project environment is to collect and use metrics observed during the course of the project, such as how much effort was expended in completing the project, how many defects were found during testing, or how many defects were found by the client following delivery of the tested system. This information subsequently can be used to determine whether the testing process is improving over time, for example, by comparing the current effectiveness in finding defects against previously observed results.

Metrics can also be used to investigate the benefits of adopting a specific approach to testing – such as the introduction of automated testing – by comparing the efficiency of the testing process before introduction with that observed following rollout of the tool. The Crown Management Systems case study (Chapter 17) provides just such an example of the role and use of metrics in a testing project environment.

If no formal, quantitative measurements are made, it is only possible to make qualitative statements about the effectiveness of the testing process. This may assure senior management in the short term, but in the long term will not help to improve the testing process.

Overview

The objective of user acceptance testing is to confirm that the application under test (AUT) meets its business requirements and to provide confidence that the system works correctly and is usable before it is formally “delivered” to the end user(s). User acceptance testing is conducted by one or more user representatives with the assistance of the test team.

User acceptance testing is considered distinct from operations acceptance testing, which is used to verify the operations and administrative aspects of the AUT (e.g., the installation of updates to the AUT, back-up, archive, and restoring the AUT and its data, and the registration of new users and assigning of their privileges). In practice, where the AUT supports simple administrative facilities, user and operations acceptance testing are often combined into a single testing exercise. Also see Chapter 10 – Operations Acceptance Testing.

User acceptance testing should employ a black box approach to testing and should make use of thread testing techniques to verify the high-level business requirements of the system. In practice, the user representative(s) will test the AUT by performing typical tasks they would perform during normal usage of the system.

User acceptance testing should also address testing of system documentation (such as user guides) by the user representative(s).

Introduction

This chapter provides an overview of testing to provide an understanding of what testing is and why it is such a challenge. It also emphasizes that whenever we test software, the process must be made as efficient and effective as possible.

Readers familiar with the need for efficient and effective testing may not need to read this chapter.

The Challenge of Testing

So, just how difficult is testing? To help answer this question, consider the following example.

Imagine we have a requirement to test a simple function, which adds two, thirty-two bit numbers together and returns the result. If we assume we can execute 1,000 test cases per second, how long will it take to thoroughly test this function?

If you guessed seconds, you are way out. If you guessed minutes, you are still cold. If you guessed hours, or days, or even weeks you are not even slightly warm. The actual figure is – 585 million years.

But surely this is a daft example. Nobody in his or her right mind would test such a function by trying out every single possible value! In practice, we would use some formal test design techniques such as boundary value analysis and equivalence partitioning to help us select specimen data for our test cases (see Chapter 3 for details of test design techniques).

Introduction

This appendix contains an example illustrating the testing technique of boundary value analysis described in Chapter 3, which the test analyst can use to select specimen data for use in testing the application under test (AUT). This example uses the same testing problem as that presented in Appendix K – Equivalence Partition Example.

The Testing Problem

The specification for a software system for validating expenses claims for hotel accommodation includes the following requirements:

• There is an upper limit of £90

• Any claims above £90 should be rejected and cause an error message to be displayed

• All expense amounts should be greater than £0 and an error message should be displayed if this is not the case.

Analyzing the Testing Requirements

To support the process of analyzing the previous requirement, it is of benefit to graphically show the boundaries and to determine the boundary values and significant values either side of the boundaries (see Figure L1).

Designing the Test Cases

The next step is to design the test cases by drawing up a table showing the test case ID, the values about and on the boundary to be input for the test, the boundary it tests, and the expected output or result of the test case (see Table L1).

Overview of the Organization

Automatic Data Processing Limited (ADP) is the leading provider of employer services in the world. Employer services in the past was simply a synonym for payroll, but nowadays it has a significantly wider definition, embracing human resources, expense management, time and attendance, recruitment services, and employer solutions based on Internet technologies. As each of these service extensions has matured, ADP has been in the vanguard of technical and business innovation, consistently redefining the market and expanding the value its services deliver to clients.

Founded in 1949, ADP has been an astonishing success story for more than half a century. Its unrivaled record of profitable growth over more than 150 consecutive quarters bears witness to the extraordinary stability of ADP, making it the partner of choice in critical aspects of a vast number of businesses. Through a single-minded commitment to providing high-quality, employment-related services such as payroll, ADP has grown steadily and now serves a client community of more than 450,000 organizations worldwide.

ADP Employer Services began its United Kingdom (U.K.) operations in 1965 and these are now available throughout the United Kingdom via ADP's purpose-built client, sales, and support centers in Staines, Manchester, Birmingham, and Chessington.

ADP's business strategy is based on the need to grow through consistent improvement and innovation in the provision of employer services and where appropriate to exploit new and emerging technologies to assist in this goal.

Overview

The objective of systems integration testing is to provide confidence that the application under test (AUT) is able to successfully interoperate with other specified software systems and does not have an adverse affect on other systems that may also be present in the live environment or vice versa. Systems integration testing is conducted by the testing team under the supervision of the test team leader. Systems integration testing may also be termed compatibility testing or simply integration testing (as termed in the British Library testing process for example – see Chapter 15).

It is possible that the testing tasks performed during systems integration testing may be combined with system testing, particularly if the AUT has little or no requirement to interoperate with other systems.

Systems integration testing should employ black box techniques and will test high-level interoperability requirements of the AUT without considering the internal construction of the AUT (e.g., testing business processes and complete transactions that require intersystem communication or interaction). The use of negative testing and error guessing techniques (see Chapter 3) are particularly appropriate during compatibility testing for uncovering unanticipated problems.

In terms of the V Model, systems integration testing corresponds to the specification phase of the software development lifecycle (see Figure 8.1).

So what is this testing thing then?

A question that I and many other aspiring software testing professionals get asked frequently. We try to explain, and watch the questioner's eyes glaze over as he or she furtively starts looking around to find a reason to step away or change the subject.

Isn't it strange that this happens? Is it that the explanation is simply incomprehensible by any sane person, or is it that software testers themselves need to understand in simple terms what it is they do?

I can add my own personal perspective on this issue; I help organize the British Computer Society Specialist Group in Software Testing (BCS SIGiST), and I am regularly surprised by two things: first, 70% of the attendees have never attended any form of networking meeting before, and second (and perhaps more important), none of the attendees read any of the many software testing books that exist.

Having had the opportunity to discuss this with many of the attendees, it is clear that the reason for this is the content of the typical testing book – it's just too complex for them to get their teeth into and understand. What is needed is a straightforward, simple-to-read, and simple-to-use testing book.

When the first edition of Testing IT was released in 2001, this changed. For once there was a book that provided a simple overview of what testing was, with straightforward guidance for test practitioners, plus a selection of easy-to-use testing templates.

Overview

The objective of unit testing is to ensure that reliable program units are produced that meet their requirements. The process is primarily intended to identify errors in program logic. Unit testing is conducted by the development team under the supervision of the development team leader. Typically, the software engineer or programmer who coded the unit will design and run a series of tests to verify that the unit meets its requirements. Unit testing may also be termed component testing.

Each unit should be tested individually and in isolation (although it may be necessary to employ a test harness) by exercising its inputs and observing its outputs. It may also be possible for the unit to be tested using the facilities available in the development environment (such as stepping through the statements of code using a debugger). Once confidence has been established in the unit, it should then be tested in collaboration with other interoperating units.

Unit testing is typically a white box testing activity, based primarily on the functional and data requirements expressed in the requirements specification for the application under test (AUT), as well as any supplementary material (such as design documentation, user guides, or prototype code).

Introduction

This appendix contains a test result record form template that may be copied and used to record the result of the individual test cases comprising a particular test phase. An electronic version of the template can be obtained using the following link: http://www.cambridge.org/9780521148016/.

The test result record form contains the following sections:

• Project ID – the unique project identifier

• AUT title – the definitive title of the application under test (front sheet only)

• AUT version – the definitive version information for the AUT (front sheet only)

• Testing phase – the title of the phase of testing being conducted (e.g., unit test, integration test, system test) (front sheet only)

• Date of test – the date on which the test was performed (front sheet only)

• The names of the tester and the test observer

• Test ID – the unique identifier for the test

• Time of test – the time at which each individual test case was started

• Observed test result – a brief textual description of the result of the test

• Test result category – the test result category assigned to this test (as specified from the test specification document for this testing phase – see Appendix D)

• Test error description – a brief textual description (if appropriate) of the observed test error

• Tester signature – the signature of the tester, recording participation in the test, observation of the result of the test, and agreement to the documented test result

• Test observer signature – the signature of the independent test observer, recording observation of the result of the test and agreement to the documented test result.

Introduction

This appendix contains an example illustrating the testing technique of state transition analysis described in Chapter 3, which the test analyst can use to select specimen data for use in testing the application under test (AUT). State transition analysis is also termed state transition testing.

The Testing Problem

This example describes part of the specification for the software controlling the operation of a water pump:

• The pump can be in one of three states: isolated, ready, or running. The pump cannot start (i.e., move to the running state) if it is isolated

• Opening the water valve will move the pump from the isolated into the ready state. Closing the water valve when the pump is ready will return it to the isolated state

• Pressing a start button when the pump is in the ready state will start the pump and move it into the running state. Pressing a stop button when the pump is running will stop the pump and move it into the ready state.

Analyzing the Testing Requirements

To support the process of analyzing the previous requirement, the test analyst draws a state transition diagram to graphically show the states, their transitions, and the events (see Figure M1).

Introduction

This appendix provides a brief overview of the process of usability testing. For a comprehensive treatment of this topic see (9).

Specifically, this appendix describes the roles and responsibilities, test design, test and post-test activities, and provides a graphical view of the organization of a typical usability test.

Roles and Responsibilities

The following roles are applicable to usability testing:

• Test team leader – the usability testing project is managed on a day-to-day basis by the test team leader, who is also responsible for liaising with all other staff involved in testing

• Test analyst – the usability testing scenarios are devised by the test analyst with reference to available usability requirements for the application under test (AUT) and in close collaboration with the development team leader. This collaboration is essential for identifying specific areas of concern within the AUT user interface (UI) that may need to be investigated

• User representative(s) – Usability testing is conducted by a number of user representatives (users). For the purposes of obtaining accurate statistical results, the more users that can be persuaded to get involved, the better. As a general rule, ten users is the minimum number that should be involved in a testing project

• Test observer – the testing is remotely monitored by a test observer, who monitors all sources of information from the test (video, audio, and captured AUT UI images) and who has computer-aided support for subsequent analysis of the information

• Development team leader – the development team leader may optionally sit with the test observer to observe the test

• […]

Overview

The objective of operations acceptance testing is to confirm that the application under test (AUT) meets its operations requirements, and to provide confidence that the system works correctly and is usable before it is formally “handed over” to the operations user(s). Operations acceptance testing is conducted by one or more operations representatives with the assistance of the test team.

Operations acceptance testing is considered distinct from user acceptance testing, which is used to verify that the AUT meets its business requirements, and to provide confidence that the system works correctly and is usable before it is formally “delivered” to the end user(s). In practice, where the AUT supports simple administrative facilities, operations and user acceptance testing are often combined into a single testing exercise. Also see Chapter 9 – User Acceptance Testing.

Operations acceptance testing should employ a black box approach to testing and should make use of thread testing techniques to verify high-level operations requirements of the system. In practice, the operations representative(s) will test the AUT by performing typical tasks they would perform during normal usage of the system.