12 - Testing Systems with Complex State

Summary

This chapter discusses various approaches to on-the-fly testing of closed systems. (We discuss on-the-fly testing of reactive systems in Chapter 16.)

The main difference between on-the-fly testing and offline testing is that in the case of on-the-fly testing, action sequences are not known beforehand; that is, there is no pregenerated test suite.

The use of on-the-fly testing is most relevant when the state space of the model is too large to be covered comprehensively. It may still be possible to use various finitization techniques discussed in earlier chapters to cope with this problem; on-the-fly testing is an alternative approach that may be more suitable in some situations, for example, when existence of a test suite is not relevant, or when it is not clear how to finitize or abstract the model.

Another case when on-the-fly testing is useful is when implementation under test (IUT) has hidden states that are not distinguished by the model. In other words, if a state is revisited in the model, the corresponding state in the implementation may be new. In such a case a test suite that provides transition coverage of the model might not expose an error because a transition in the model has to be traversed multiple times before the IUT is in a state where the error occurs. The error discussed in Chapter 2 is an example.

The key concept of on-the-fly testing is a strategy. Intuitively, a strategy is a function that during each step of testing selects which action to apply next.