Introduction

The following pages of this appendix contain a test result record form template (Figure F.1) that may be copied and used within your own testing task or project.

The test result record form template contains the following sections:

• Project ID – the unique project identifier;

• AUT Title – the definitive title of the Application Under Test (front sheet only);

• AUT Version – the definitive version information for the AUT (front sheet only);

• Iteration ID – the unique identifier for the iteration this test is being conducted in (front sheet only);

• Date of Test – the date on which the test was performed (front sheet only);

• Test ID – the unique identifier for the test;

• Time of Test – the time at which each individual test case was started;

• Observed Test Result – a brief textual description of the result of the test;

• Test Result Category – the test result category assigned to this test (as specified from the Test Specification document for this testing phase; see Appendix D);

• Test Error Description – a brief textual description (if appropriate) of the observed test error;

• Name of the Tester;

• Tester Signature – the signature of the tester, recording their participation in the test, their observation of the result of the test, and their agreement to the documented test result;

• Name of the Test Observer – this might be the user/customer representative, or somebody from the quality assurance group (if your organization has one); and

• Test Observer Signature – the signature of the test observer, recording their observation of the result of the test, and their agreement to the documented test result.

Introduction

My name is Nick Sewell and I work for Ivar Jacobson Consulting Ltd. as the European Managing Director. I have over twenty years' commercial experience in the software industry – twelve of these ensuring that software teams deliver software successfully using a range of techniques.

In a fast-moving customer environment where requirements are changing all the time to support the user needs, the testing strategy has to be superagile.

For example, I have recently been working as a part of a team enhancing a large business critical information system that needs to be updated every six weeks to keep pace with an ever-changing variety of information sources and analytical needs.

Introduction

My name is James Wilson and I am the CEO of Trinem, based in Edinburgh, United Kingdom. I cofounded Trinem with Philip Gibbs in 2001 after working for a number of years as an IT consultant, primarily delivering software configuration management (SCM) services and solutions. Since 2001, Trinem has implemented SCM projects that have inherited various forms of technologies, methods, and scale.

Trinem has implemented processes, and had their technology implemented, for a wide spectrum of organizations, including NASA (United States), Bank of New York (United Kingdom), HBOS (United Kingdom), Belastingdienst (Holland), Syscom (Taiwan), SDC (Denmark), and GE (United Kingdom and Holland).

This case study illustrates the challenges, approaches, and benefits that I have encountered while developing agile application development processes.

This case study will also explore some of the key technologies that were used, both commercial and open source, to augment the implementations delivered by Trinem and myself.

Agile, Extreme Programming, Scrum, and/or Iterative?

Invariably when customers ask for consulting services to assist with the implementation of SCM and application development best practices, they already have an idea of what is happening in the industry.

Introduction

This appendix reviews the essential rules and practices of Extreme Programming (XP), focusing in more detail on the quality aspects of the method.

The XP rules and practices are grouped into four sections: (1) planning, (2) designing, (3) coding, and (4) testing. It is notable that in virtually every description of the principles of XP you read, you will find that testing is covered last. In this appendix, testing is covered as the first topic.

Testing

Quality assurance (QA) is considered to be an essential aspect of XP. QA may be performed by a separate group, or it can be integrated into the development team itself. In either case, XP promotes the idea that the developers have a high degree of contact with the QA group; this is very much in line with the XP philosophy of closer communications between the stakeholders in a project.

XP provides some excellent guidance on testing:

• XP encourages early and frequent testing.

• XP looks to make the process as efficient and effective as possible by promoting the use of test automation.

• XP demands that all code have associated tests, and that where code is changed that the tests be rerun (and/or modified where appropriate to reflect the changes).

• XP requires test design to be completed before coding has started, with all code having associated unit tests.

There are just two types of test in XP:

• Unit tests written by the developers to test their code. These are white box tests [4], which rely on the knowledge of the code and its purpose.

• […]

Introduction

My name is Geoff Thompson. I have been involved in software testing for nearly twenty years. In addition to automation, I have in my time been a test analyst right through to a test program manager. In that time I have experienced or directly used many life-cycle delivery approaches, such as waterfall [5], Timeboxes [21], V-model [4], Rational Unified Process [7], Interactive [27], Agile and Scrum [26].

I am currently the Services Director for Experimentus Ltd, a U.K.-based software quality consultancy, specializing in process improvement, project management, test, configuration management, and requirements management.

I am a founding member of the Information Systems Examination Board Software Testing Board and also a founding member of the International Software Testing Qualifications Board [42], and am currently the U.K. representative to the Board. I am also the founder and chairman of the U.K. Testing Board (www.uktb.org.uk).

The following case study describes a project for which I was the test program manager for an FTSE 100 life assurance company (the BigLifeAssurance Insurance Co. Ltd [BLA Ltd]) that was delivered using waterfall/V-model and agile approaches alongside each other in separate but dependent projects.

Introduction

This chapter draws upon the information provided in the cases studies and their analyses to make a number of proposals for how you might set up and run your own practical, effective, and efficient agile testing process. Using the set of agile testing practices highlighted during the analyses, this chapter provides a series of proposals, allowing you to pick and mix from these practices to create an agile method tailored to your own particular development and testing requirements.

In providing a valuable source of real-world development and testing practices, the case studies reinforce the idea that each project is unique. Each project is of different size and complexity and, in reviewing the agile practices, you need to consider what will work for you. To simplify this task, the practices in this chapter are organized around four stereotypical projects, and you should make an assessment about which one best matches your own agile requirements. Try not to use the information provided dogmatically; look at the proposed agile practices critically and decide if you think they would work for you.

As a final thought, whatever set of practices you pick will probably not be perfect, so make sure you follow the advice provided on process improvement: keep doing what works for you and challenge the use of, or modify, those practices that do not.

Introduction

My name is Martin Phillips and I work for IBM. I've been in software development for almost nine years, working mostly on functional testing of “off-the-shelf” middleware. This software was developed primarily using the waterfall model [5] (with twelve- to eighteen-month cycles), though more recently projects have been more iterative (with two- to three-month iterations).

For the past six months I've been the test lead for a newly formed agile development team, and am responsible for delivering a new middleware product for an emerging market.

This case study describes the challenges we have faced testing the product and what lessons we have learned.

Overview of the Testing Challenge

IBM is very aware of the commercial value of emerging markets; like many companies, it wants to get a foothold in these new areas quickly, but with products that meet the high expectations that customers have of IBM.

Developing with an agile approach enables our team to develop a functional product with early and frequent beta releases.

Introduction

My name is Michael G. Norman. I have been involved with software test and performance for more than ten years, most recently having taken on the role as CEO of Scapa Technologies Ltd, a software tools vendor company. I was formerly involved in forming the Eclipse Test and Performance Tools Project (TPTP, formerly Hyades), which I ran in collaboration with colleagues from IBM and Intel and served on its Project Management Committee and the Board of Eclipse for around five years.

Scapa Technologies is involved with a range of testing engagements with its several hundred enterprise customers and tries to be methodologically neutral in the way it develops its tooling, making it applicable in both traditional and agile environments. In cases where it performs its own services, Scapa adopts a variation on the agile approach, which is tailored toward its predominant customer profile: integration and deployment of packaged applications by or on behalf of enterprise customers.

This case study describes a project where Scapa was engaged to test and produce very fast feedback on a set of applications that were being deployed by a telecommunications vendor via a remote desktop technology.

Aims

The aim of this chapter is to show how systems with a “persistent state” can be modelled. On completion of the chapter, the reader should be able to develop models of systems which contain persistent state components. The difference between this style and the functional modelling style used so far in this book will be highlighted by revisiting the explosives controller and the trusted gateway examples.

Introduction

Using formal modelling techniques, computing systems may be described at many different levels of abstraction. The models presented so far in the book have been set at a relatively high level of abstraction. This is reflected both in the data types used and in the way functionality is described through mathematical functions that take some data representing the system as input parameters and return a result which describes the system after the computation has been performed. In some cases, the function can be described without explicitly constructing its result (e.g. see Subsection 6.4.3).

This functional modelling style has its limitations. Few computing systems are actually implemented via pure functions. More often, they have variables that hold data which are modified by operations invoked by some outside user. These variables are persistent in the sense that they continue to hold data between operation invocations. If the purpose of a model is to document design decisions about the split between ordinary parameters to functions and persistent variables it is necessary to use operations in VDM-SL. These operations take inputs and return results, but they also have some effect (often called a side-effect) on the persistent variables.

Aims

The aim of this chapter is to show how relationships between data can be modelled as mappings. The mapping type constructor and operators in VDMSL are introduced through an example from the nuclear industry. On completing this chapter, the reader should be confident in modelling and analysing systems involving mappings.

Introduction

Computing systems frequently centre on relationships between sets of values. For example, a database might link a set of customer identifiers to detailed information. Such relationships can often be modelled as mappings from elements of one set, known as the domain, to elements of the other set, known as the range. Mappings can be thought of as tables in which one can look up the domain element and read across to see the range element to which it is related. We will say that each domain element maps to the corresponding range element. Each line of the table, being a small part of the mapping, is called a maplet. Each domain element can have only one maplet in a mapping, so there is no ambiguity about which range element it points to. For example, the following table represents a mapping from names (strings of characters) to bank balances (integers).

Aims

This final chapter concerns the use of VDM in industrial practice. We aim to equip the reader to apply VDM technology cost-effectively in industrial software development processes, and to stay abreast of the state of the art in VDM and formal modelling. We aim to introduce the contribution that formal modelling can make to the tasks that are at the core of commercial development processes. We will illustrate this with several real industrial applications of VDM. Finally, we aim to provide information on the recent extensions to VDM and VDMTools, and how to gain the most from the VDM and formal methods communities.

Introduction

Modelling in a formal language is not a panacea for every problem in system and software development but, if used thoughtfully, it can yield significant benefits. The deciding factor in using VDM technology (the combination of VDM-SL and VDMTools) has to be cost-effectiveness. The cost of developing a system model during the early stages of design should be recouped when the improved understanding of system functionality reduces the reworking required to deal with defects that are uncovered during later activities such as testing and maintenance. In this chapter we discuss a range of software development activities, some of the problems that can arise during their execution and the ways in which the use of VDM can address some of these (Sections 13.2 to 13.4). Some hints on how to start using VDM are presented (Section 13.5). We illustrate the approach by describing recent industrial applications of VDM and its extended forms (Sections 13.6 and 13.7).

Aims

In this chapter, we aim to provide an awareness of the issues involved in constructing and analysing large-scale models. We will introduce modular structuring facilities in VDM-SL as an illustration of the features required in a structuring mechanism. On completion of this chapter, the reader should be able to exploit modular structuring and the potential for re-use in large models.

Introduction

In any course on modelling, one is naturally limited in the size of model which can be developed and presented. However, in any realistic application of modelling technology, questions of scale must be addressed. How can one manage the complexity of developing and analysing a model which contains many related parts?

Before answering this question, it is worth remembering that models should be kept as simple as possible while still capturing the aspects of the system which are felt to be relevant to the analysis. Careful use of abstraction means that many systems can be usefully modelled without encountering problems of scale. However, for some applications, particularly where the product is safety-related, a formally defined language such as VDM-SL must be applied to the production of a substantial model, and so the management of the model's size and complexity becomes a significant issue.

In programming languages, the management of complexity has led to the adoption of modular structuring mechanisms for programs, and this approach has also been applied to VDM-SL. All the models presented so far have been flat in the sense that they have consisted of a series of definitions in a single document.

Aims

The aim of this chapter is to introduce the use of logic for stating the properties of data and functions in system models. The logic used in VDM-SL models is introduced via a temperature monitor example. On reaching the end of the chapter, the reader should be able to state and analyse logical expressions in VDMTools Lite.

Introduction

An important advantage of building a model of a computing system is that it allows for analysis, uncovering misunderstandings and inconsistencies at an early stage in the development process. The discovery of a possible failure of the Expert To Page function in the previous chapter was the result of just such an analysis. The ability to reason about the types and functions in a model depends on having a logic (a language of logical expressions) in which to describe the properties of the system being modelled and in which to conduct arguments about whether those properties hold or not.

This chapter introduces the language of logical expressions used in VDMSL, based on Predicate Logic. It begins by introducing the idea of a predicate, then examines the basic operators which allow logical expressions to be built up from simpler expressions. Finally, we examine the mechanisms for dealing with mis-application of operators and functions in the logic of VDM-SL.

The temperature monitor

The example running through this chapter continues the chemical plant theme. Suppose we are asked to develop the software for a temperature monitor for a reactor vessel in the plant. The monitor is connected to a temperature sensor inside the vessel from which it receives a reading (in degrees Celsius) every minute.

Aims

Recursive structures are common in many applications, notably in computer language processing. They present particular modelling challenges and so we devote a chapter to them. The aim here is to show how recursive data structures such as trees and graphs are defined and used through recursive traversal. We do not introduce new VDM language concepts at this stage, but consolidate the reader's knowledge and experience by showing how VDM copes with this important class of system. We add further abstraction lessons by considering the executability of functions.

Recursive data structures: trees

Recursion arises in many significant computing applications. In Chapter 7 we introduced recursive functions as a means of traversing collections of values, and illustrated their use on sequences. However, recursion is also central to an understanding of other data structures, including trees and graphs, that arise in many significant computing applications. This chapter explores the modelling of such recursive structures further. We begin by examining tree structures and illustrate their use by examining abstract syntax trees – an application area that underpins many applications in design and programming support environments, including VDMTools. We go on to examine more general graph structures, using an application from machine code optimisation to illustrate recursive traversal. The abstraction lesson in this chapter concerns the costs and benefits of executable models.

We begin by considering a common data structure: a tree. A tree is a collection of points (termed nodes) connected to each other by links (termed arcs).