In this chapter, we explore, in more detail, the software development methodology that is used with CDTs. It is a methodology based on transformation. Many of the transformations that are useful for list programming were already known informally in the Lisp community, and more formally in the APL and functional programming community. The chief contributions of the categorical data type perspective are:

• a guarantee that the set of transformation rules is complete (which becomes important for more complex types); and

• a style of developing programs that is terse but expressive.

This style has been extensively developed by Bird and Meertens, and by groups at Oxford, Amsterdam, and Eindhoven. A discussion of many of the stylistic and notational issues, and a comparison of the Bird–Meertens approach with Eindhoven quantifier notation, can be found in [17]. Developments in the Bird-Meertens style are an important interest of IFIP Working Group 2.1.

An Integrated Software Development Methodology

A software development methodology must handle specifications that are abstract, large, and complex. The categorical data type approach we have been advocating plays only a limited role in such a methodology because it is restricted (at the moment) to a single data type at a time. Although it is useful for handling the interface to parallel architectures, it is too limited, by itself, to provide the power and flexibility needed for large application development.

We have shown how to build categorical data types for the simple type of concatenation lists. In this chapter we show the data type construction in its most general setting. While there is some overhead to understanding the construction in this more general setting, the generality is needed to build much more complex types. We illustrate this in subsequent chapters by building types such as trees, arrays, and graphs.

More category theory background is assumed in this chapter. Suitable references are [127,158].

Categorical Data Type Construction

The construction of a categorical data type is divided into four stages:

1. The choice of an underlying category of basic types and computations on them. This is usually the category Type, but other possibilities will certainly be of interest.

2. The choice of an endofunctor, T, on this underlying category. The functor is chosen so that its effect on the still-hypothetical constructed type is to unpack it into its components. Components are chosen by considering the type signatures of constructors that seem suitable for the desired type. When this endofunctor is polynomial it has a fixed point; and this fixed point is defined to be the constructed type.

3. The construction of a category of T-algebras, T-AIg, whose objects are algebras of the new type and their algebraic relatives, and whose arrows are homomorphisms on the new type. The constructed type algebra (the free type algebra) is the initial object in this category. The unique arrows from it to other algebras are catamorphisms.

4. […]

The central theme of this book is that the structure of a computation on a data type reflects the structure of the data type. This is true in two senses:

• Any homomorphism on a data type is intimately related to the algebraic structure of its codomain; which can be exploited in the search for programs, and

• The evaluation of any homomorphism can follow the structure of its argument; which can be exploited in computing programs.

Structured data types and the homomorphisms on them, called catamorphisms, form a programming model for parallel computation that has many attractive properties.

There is a desperate need for a model of parallel computation that can decouple software from hardware. This decoupling occurs in two dimensions: decoupling the rate of change of parallel hardware (high) from that of parallel software (low, if it is to be economic); and decoupling the variety of parallel hardware from a single, architecture-independent version of the software.

Such a model is hard to find because the requirements are mutually in tension. A model must be opaque enough to hide target architectures and the complexity of parallel execution, while providing a semantic framework that is rich enough to allow software development. At the same time, it must be partly translucent so that the costs of programs can be visible during development, to allow intelligent choices between algorithms.

In this chapter we build a much more complex type, the type of arrays. The construction of arrays as a categorical data type is significantly different and more complex than the constructions seen so far, so this chapter illustrates new aspects of the construction technique.

Arrays are an important type because of the ubiquitous use of Cartesian coordinate systems and the huge edifice of linear algebra built on top of them. Almost all scientific and numeric computations require arrays as central data structures.

While the need for a data type of arrays is undisputed, there has always been some disagreement about exactly what arrays should represent: should the entries of an array all be of the same type (homogeneous) or might they be different (inhomogeneous); should extents be the same in each dimension (rectangular) or might they differ (ragged); are arrays of different sizes members of the same type or of different types. Different programming languages have answered these questions differently.

Arrays appeared early in languages such as Fortran, which had homogeneous, rectangular arrays but was ambivalent about how they should be typed. Arrays had to be declared with their shapes, and shapes of arguments and parameters had to agree (although some of these rules were relaxed later). Fortran even went so far as to reveal the storage allocation of arrays (by columns) at the language level.

In this chapter, we explore the constraints imposed on models by the properties of parallel architectures. We are only concerned, of course, about theoretical properties, because we cannot predict technological properties very far into the future. Recent foundational results, particularly by Valiant [200], show that arbitrary parallel programs can be emulated efficiently on certain classes of parallel architectures, but that inefficiencies are unavoidable on others. Thus a model of parallel computation that expresses arbitrary computations cannot be efficiently implementable over the full range of parallel architecture classes. The difficulty lies primarily in the volume of communication that takes place during computations. Thus we are driven to choose between two quite different approaches to designing models: accepting some inefficiency, or restricting communication in some way.

Parallel Architectures

We consider four architecture classes:

• shared-memory MIMD architectures, consisting of processors executing independently, but communicating through a shared memory, visible to them all;

• distributed-memory MIMD architectures, consisting of processors executing independently, each with its own memory, and communicating using an interconnection network whose capacity grows as p log p, where p is the number of processors;

• distributed-memory MIMD architectures, consisting of processors executing independently, each with its own memory, and communicating using an interconnection network whose capacity grows only linearly with the number of processors (that is, the number of communication links per processor is constant);

• SIMD architectures, consisting of a single instruction stream, broadcast to a set of data processors whose memory organisation is either shared or distributed.

So far we have discussed the properties that a model of parallel computation ought to have and have claimed that models built from categorical data types have these properties. In this chapter we show how to build a simple but useful categorical data type, the type of join or concatenation lists, and illustrate its use as a model. We show how such a model satisfies the requirements, although some of the details are postponed to later chapters.

The language we construct for programming with lists is not different from other parallel list languages in major ways in the sense that most of the list operations are familiar maps, reductions, and prefixes. The differences are in the infrastructure that comes from the categorical data type construction: an equational transformation system, a deeper view of what operations on lists are, and a style of program development. When we develop more complex types, the construction suggests new operations that are not obvious from first principles.

For the next few chapters we concentrate on aspects of parallel computation on lists. We describe the categorical data type construction in more detail in Chapter 9 and move on to more complex types. The next few sections explain how to build lists in a categorical setting. They may be skipped by those who are not interested in the construction itself. The results of the construction and its implications are summarised in Section 5.5.

We have already discussed why a set of cost measures is important for a model of parallel computation. In this chapter we develop something stronger, a cost calculus. A cost calculus integrates cost information with equational rules, so that it becomes possible to decide the direction in which an equational substitution is cost-reducing. Unfortunately, a perfect cost calculus is not possible for any parallel programming system, so some compromises are necessary. It turns out that the simplicity of the mapping problem for lists, thanks to the standard topology, is just enough to permit a workable solution.

Cost Systems and Their Properties

Ways of measuring the cost of a partially developed program are critical to making informed decisions during the development. An ideal cost system has the following two properties:

1. It is compositional, so that the cost of a program depends in some straightforward way on the cost of its pieces. This is a difficult requirement in a parallel setting since it amounts to saying that the cost of a program piece depends only on its internal structure and behaviour and not on its context. However, parallel operations have to be concerned about the external properties of how their arguments and results are mapped to processors since there are costs associated with rearranging them. So, for parallel computing, contexts are critically important.

2. It is related to the calculational transformation system, so that the cost of a transformation can be associated with its rule.

In this chapter we define the categorical data types of graphs. Graphs are ubiquitous in computation, but they are subtly difficult to work with. This is partly because there are many divergent representations for graphs and it is hard to see past the representations to the essential properties of the data type.

We follow the now familiar strategy of defining constructors and building graphs as the limit of the resulting polynomial functor.

Graphs have a long history as data structures. Several specialised graph languages have been built (see, for example, [69]), but they all manipulate graphs using operations that alter single vertices and edges, rather than the monolithic operations we have been advocating.

An important approach to manipulating graphs is graph grammars. Graph grammars [71] are analogues of grammars for formal languages and build graphs by giving a set of production rules. Each left hand side denotes a graph template, while each right hand side denotes a replacement. The application of a rule occurs on a subgraph that matches the left hand side of some rule. The matching subgraph is removed from the graph (really a graph form) and replaced by the right hand side of the rule. Various different conventions are used to add edges connecting the new graph to the rest of the original graph. Graph grammars are primarily used for capturing structural information about the construction and transformation of graphs. They do not directly give rise to computations on graphs.

The second substantially technical issue in the design of parallel computers concerns the complexity of each of the processing elements. To a very significant extent this will reflect the target functionality of the system under consideration. If a parallel computer is intended for use in scientific calculations, then in all likelihood floating-point operations will be a sine qua non and appropriate units will be incorporated. On the other hand, a neural network might be most efficiently implemented with threshold logic units which don't compute normal arithmetic functions at all. Often, the only modifier on such a natural relationship will be an economic or technological one. Some desirable configurations may be too expensive for a particular application, and the performance penalty involved in a non-optimum solution may be acceptable. Alternatively, the benefits of a particularly compact implementation may outweigh those of optimum performance. There may even be cases where the generality of application of the system may not permit an optimum solution to be calculated at all. In such cases, a variety of solutions may be supportable.

We will begin our analysis of processor complexity by considering the variables which are involved, and the boundaries of those variables.

Analogue or digital?

Although almost all modern computers are built from digital circuits, this was not always, and need not necessarily be, so. The original, overwhelming reason for using digital circuits was because of the vastly improved noise immunity which could be obtained, particularly important when using highprecision numbers.

In order to make sense of the way in which users control parallel computers, we shall have to adopt a rather wider definition of the idea of programming than that which is usually taken. This is principally because of the existence of systems which are trained rather than programmed. Later in this chapter I shall attempt to show the equivalence between the two approaches but it is probably best to begin by considering how the conventional idea of programming is applied to parallel systems.

Parallel programming

There are three factors which we should take into account in order to arrive at a proper understanding of the differences between one type of parallel language and another and to appreciate where the use of each type is appropriate. These are whether the parallelism is hidden or explicit, which paradigm is employed and the level of the language. Although there is, inevitably, a certain amount of overlap between these factors, I shall treat them as though they were independent.

The embodiment of parallelism

There is really only one fundamental choice to be made here – should the parallelism embodied in a language be implicit or explicit? That is, should the parallelism be hidden from the programmers, or should it be specified by them? The first alternative is usually achieved by allowing the use of data types, in a program, which themselves comprise multiple data entities.

The purpose of this book has been to introduce the reader to the subject of parallel computing. In attempting to make the subject digestible, it is inevitable that a great deal of advanced material has been omitted. The reader whose interest has been kindled is directed to the bibliography, which follows this chapter, as a starting point for continued studies. In particular, a more rigorous treatment of advanced computer architectures is available in Advanced Computer Architectures by Hwang.

It should also be noted that parallel computing is a field in which progress occurs at a prodigious rate. So much work is being done that some startling new insight or technique is sure to be announced just after this book goes to press. In what follows, the reader should bear in mind the developing nature of the field. Nevertheless, a great deal of material has been presented here, and it is worthwhile making some attempt to summarise it in a structured manner.

I have attempted to make clear in the preceding chapters that understanding parallel computing is an hierarchical process. A valuable degree of insight into the subject can be obtained even at the level considered in the first chapter, where three basic classes or types of approach were identified. Thereafter, each stage of the process should augment the understanding already achieved. It is up to each reader to decide on the level of detail required.

As was indicated in Chapter 1, there is a prima facie case for supposing that parallel computers can be both more powerful and more cost-effective than serial machines. The case rests upon the twin supports of increased amount of computing power and, just as importantly, improved structure in terms of mapping to specific classes of problems and in terms of such parameters as processor-to-memory bandwidth.

This chapter concerns what is probably the most contentious area of the field of parallel computing – how to quantify the performance of these allegedly superior machines. There are at least two significant reasons why this should be difficult. First, parallel computers, of whatever sort, are attempts to map structures more closely to some particular type of data or problem. This immediately invites the question – on what set of data and problems should their performance be measured? Should it be only the set for which a particular system was designed, in which case how can one machine be compared with another, or should a wider range of tasks be used, with the immediate corollary – which set? Contrast this with the accepted view of the general-purpose serial computer, where a few convenient acronyms such as MIPS and MFLOPS (see Section 6.1.3) purport to tell the whole story. (That they evidently do not do so casts an interesting sidelight on our own problem.)

The second reason concerns the economic performance, or cost-effectiveness, of parallel systems.

Up to this point we have considered the subject of parallel computing as a series of almost separated facets – paradigms, languages, processor design, etc. – each of which can be examined in isolation. Of course, in arriving at the design of any real system, be it commercial or prototype, this approach is very far from the true manner of proceeding. In real system design, the watchword is usually compromise – compromise between specific and general applicability, compromise between user friendliness and efficiency in programming, compromise between the demands of high performance and low cost. No imaginary design exercise can do justice to the complexity or difficulty of this process, because the constraints in imaginary exercises are too flexible. In order to see how the various design facets interact, we need to examine real cases. This is the purpose of this chapter.

In it, I present a series of short articles concerning specific machines, each of which has been contributed by an author intimately concerned with the system in question. The brevity has been occasioned by the limited space available in a book of this sort, but it has had the beneficial effect of ensuring that the authors have concentrated on those areas which they consider to be important. Again, because of limited space I have curtailed the number of these contributions, although this has meant that some aspects of the subject remain unconsidered.