Introduction

In our world of rapid technological change, occasionally it is instructive to contemplate how much has altered in the last few years. Remembering life without the ability to view the World Wide Web (WWW) through browser windows will be difficult, if not impossible, for less “mature” readers. Is it only seven years since YouTube first appeared, a Web site that is now ingrained in many facets of modern life? How did we survive without Facebook all those (actually, about five) years ago?

In 2010, various estimates put the amount of data stored by consumers and businesses around the world in the vicinity of 13 exabytes, with a growth rate of 20 to 25 percent per annum. That is a lot of data. No wonder IBM is pursuing building a 120-petabyte storage array. Obviously there is going to be a market for such devices in the future. As data volumes of all types – from video and photos to text documents and binary files for science – continue to grow in number and resolution, it is clear that we have genuinely entered the realm of data-intensive computing, or as it is often now referred to, big data.

Interestingly, the term “data-intensive computing” was actually coined by the scientific community. Traditionally, scientific codes have been starved of sufficient compute cycles, a paucity that has driven the creation of ever larger and faster high-performance computing machines, typically known as supercomputers. The Top 500 Web site shows the latest benchmark results that characterize the fastest supercomputers on the planet.

Introduction and Background

The MapReduce programming model has had a transformative impact on dataintensive computing, enabling a single programmer to harness hundreds or thousands of computers for a single task and get up and running in a matter of hours. Processing with thousands of computers require a different set of design considerations dominate: I/O scalability, fault tolerance, and flexibility rather than absolute performance. MapReduce, and the open-source implementation Hadoop, are optimized for these considerations and have become very successful as a result.

It is difficult to quantify the popularity of the MapReduce framework directly, but one indication of the uptake is the frequency of the search term. Figure 8.1 illustrates the search popularity for terms “mapreduce” and “hadoop” over the period 2006 to 2012. We see a spike in popularity for the term “mapreduce” in late 2007, but more or less constant popularity since. For the term “hadoop,” however, we see a steady increase to about twelve times that of “mapreduce.”

These data suggest that MapReduce and Hadoop are generating interest, as seen from the number of downloads, successful startups [12, 19, 47], projects [41, 53, 57], and interest from the research community [15, 18, 62, 63, 72, 78]. These data suggest a significant increase in interest in both MapReduce and Hadoop.

The MapReduce framework provides a simple programming model for expressing loosely coupled parallel programs by providing two serial functions, Map and Reduce. The Map function processes a block of input producing a sequence of (key, value) pairs, while the Reduce function processes a set of values associated with a single key.

Data management is the organization of information to support efficient access and analysis. For data-intensive computing applications, the speed at which relevant data can be accessed is a limiting factor in terms of the size and complexity of computation that can be performed. Data access speed is impacted by the size of the relevant subset of the data, the complexity of the query used to define it, and the layout of the data relative to the query. As the underlying data sets become increasingly complex, the questions asked of it become more involved as well. For example, geospatial data associated with a city is no longer limited to the map data representing its streets, but now also includes layers identifying utility lines, key points, locations, and types of businesseswithin the city limits, tax information for each land parcel, satellite imagery, and possibly even street-level views. As a result, queries have gone from simple questions, such as, “How long is Main Street?,” to much more complex questions such as, “Taking all other factors into consideration, are the property values of houses near parks higher than those under power lines, and if so, by what percentage?” Answering these questions requires a coherent infrastructure, integrating the relevant data into a format optimized for the questions being asked.

Data management is critical to supporting analysis because, for large data sets, reading the entire collection is simply not feasible. Instead, the relevant subset of the data must be efficiently described, identified, and retrieved. As a result, the data management approach taken effectively defines the analysis that can be efficiently performed over the data.

Discovering Biological Mechanisms through Exploration

The availability of massive amounts of data in biological sciences is forcing us to rethink the role of hypothesis-driven investigation in modern research. Soon thousands, if not millions, of whole-genome DNA and protein sequence data setswill be available thanks to continued improvements in high-throughput sequencing and analysis technologies. At the same time, high-throughput experimental platforms for gene expression, protein and protein fragment measurements, and others are driving experimental data sets to extreme scales. As a result, biological sciences are undergoing a paradigm shift from hypothesisdriven to data-driven scientific exploration. In hypothesis-driven research, one begins with observations, formulates a hypothesis, then tests that hypothesis in controlled experiments. In a data-rich environment, however, one often begins with only a cursory hypothesis (such as some class of molecular components is related to a cellular process) that may require evaluating hundreds or thousands of specific hypotheses rapidly. This large number of experiments is generally intractable to perform in physical experiments. However, often data can be brought to bear to rapidly evaluate and refine these candidate hypotheses into a small number of testable ones. Also, often the amount of data required to discover and refine a hypothesis in this way overwhelms conventional analysis software and hardware. Ideally advanced hardware can help the situation, but conventional batch-mode access models for high-performance computing are not amenable to real-time analysis in larger workflows. We present a model for real-time data-intensive hypothesis discovery process that unites parallel software applications, high-performance hardware, and visual representation of the output.

I have been programming in Fortran for more than 25 years, first in FORTRAN IV and somewhat later in FORTRAN 77. In the last decade of the 20th century, I attended, together with a number of colleagues, a course on Fortran 90, given by the late Jan van Oosterwijk at the Technical University of Delft. It was also around this time that I came to know the comp.lang.fortran newsgroup, and I have learned a lot by participating in that friendly community.

In a way, I am a typical Fortran programmer. My background is physics and I learned the task of programming partly during my study, but mostly on the job. In other ways, I am not because I took a fancy to the more esoteric possibilities of programming in general and sought means to apply them in Fortran. I also began writing articles for the ACM Fortran Forum. These articles are the groundwork for this book.

This book will not teach you how to program in Fortran. There are plenty of books dedicated to that ([22], [65]). Instead, the purpose of this book is to show how modern Fortran can be used for modern programming problems, such as how techniques made popular in the world of object-oriented languages like C++ and Java fit neatly into Fortran as it exists today. It even shows some techniques for solving certain programming problems that are not easily achieved in these languages.

Interaction with the user via graphical presentation or graphical interfaces is an extremely important aspect of computing. It is also an area that depends heavily on the available hardware and the operating system. While much effort has been put into the various operating environments to hide the hardware aspect, programming a graphical user-interface (GUI) on MS Windows is completely different from doing so on Linux or Mac OS X.

Hardly any programming language defines how you can display results graphically or how to build a graphical user-interface. The common approach is to use a library that hides – if possible or desirable – the specifics of the operating system, so that a portable program results. Some libraries, however, have been designed to take advantage of exactly one operating system, so that the program fits seamlessly in that environment, at the cost of not being portable anymore. At the same time, you should not underestimate the effort and skills required for designing and building a useful and usable GUI [3].

For a Fortran programmer, the situation is a bit complicated: most GUI libraries are written with C, C++, and similar languages in mind. Furthermore, a GUI for a long-running computation should have different properties than one for filling in a database form. This chapter examines a variety of solutions.

Plotting the Results

The first type of graphical interaction to examine is the presentation of the results of a computation. The responses from the user are simple: leaf through the collection of plots – in many cases sequentially – and maybe change a display parameter here and there.

Computer programs sometimes exhibit unexpected behavior. They may function correctly for years and then, suddenly, they fail. Computer programs that solve numerical problems are no exception. The unexpected behavior, perhaps a division by zero, is often the consequence of input that does not conform to the hidden assumptions and restrictions ofthe program. For example, a Gauss elimination procedure to solve systems oflinear equations requires the matrix to be non-singular. If the matrix is numerically singular or ill-conditioned, the mathematical problem of solving the system is still well-defined, but the numerical counterpart may have a hard time delivering a decent approximation to the exact solution. The conclusion that the matrix is ill-conditioned is only drawn while solving the system.

Sometimes there is a mismatch between the specification and the actual implementation. The part of the program that reads the data accepts input that conforms to the specification, but there are a few tacit assumptions, like line-endings for a text file containing the coefficients of the matrix for the Gauss elimination program.

Subtle bugs may lurk in an otherwise well-written and well-behaved program. Can you avoid them altogether? The unsatisfying answer is probably not, but that does not mean you cannot make an effort to understand the source of these bugs. That is the intention of this chapter – to examine three simple mathematical algorithms and their implementation in Fortran.

The idea is that a straightforward implementation contains all manners of hidden assumptions about the problem that needs solving.

Quite often the public interface of a library that you are interested in is written in a different programming language than the one you use. You will need an interface library that bridges the gap between both languages. This is the case with SQLite, a lightweight database management system [44]. It is written in C and there is a host of libraries to communicate with SQLite in all manner of languages.

For Fortran, there is the fsqlite library that I developed. It was inspired by the work of Al Danial [27], but focused on FORTRAN 77, that was mainly an example of how you could interface to the SQLite library. I wanted a generic solution instead.

The design decisions and the implementation of fsqlite illustrate a more general question: how to combine Fortran and other programming languages? Each time you need to consider:

1. ■ Low-level aspects of the combination:

2. - How do the basic types of the two languages relate to each other?

3. - What are the naming and calling conventions?

4. ■ High-level aspects relating to the actual library: What is a good “model” for the interface? Should you simply write wrapping routines that merely translate the routine interfaces or should you build a higher-level interface, molding the original interface into something that is a better fit for the library's use?

This chapter will cover both types of issues in the context of interfacing with C (an intermediate layer in C is often a convenient way to communicate with any other language) and of the interface to SQLite.

DISLIN is owned by the Max Planck Institute, for Solar System Research, Lindau, Germany.

Doxygen is an open-source program maintained by Dimitri van Heesch.

GNU, gcc, and gfortran are trademarks ofthe Free Software Foundation.

gnuplot is a plot program developed and maintained by the Free Software Foundation.

Intel Fortran is a trademark of Intel.

Interacter is a trademark of Interactive Services Ltd.

Java is a trademark of Oracle.

Linux is a trademark handled by the Linux Mark Institute.

Mac OSX is a trademark of Apple.

MATLAB is a trademark of The MathWorks, Inc.

McCabe IQ is a trademark ofMcCabe Software.

MS Visual Studio and MS Windows are trademarks of MicroSoft.

MySQL is an open source project at http://www.mysql.com.

Perl is a dynamic programming language. Its home page is http://www.perl.org.

PGI CUDA is a trademark of The Portland Group and NVIDIA.

PLplot is an open source project at http://plplot.sf.net.

PostScript and PDF are trademarks of Adobe.

Python is a dynamic programming language. Its home page is http://www.python.org.

ROBODoc is an open-source program maintained by Frank Slothouber.

SQLite is an open-source database management, residing at http://www.sqlite.org.

Tcl is a dynamic programming language developed and maintained by the Tcl Association. Tk is a user-interface library that can be used with Tcl and other languages. The home page for both is http://www.tcl.tk.

Xeffort is developed and maintained by Jugoslav Dujic.

The subject of this chapter is a simple framework for automating one particular aspect of testing: unit tests. Unit tests focus on “small” parts of a program, typically single subroutines or functions. The idea is these tests give confidence in the correct operation of routines. As the code for these tests should be considered part of the program, you can repeatedly run them during development and maintenance of the whole system. Some development methods take this to the extreme. Test-driven development (TDD), for instance, prescribes that the tests be written first, before you start implementing the actual code.

This is just one way to develop a program, but thinking about the ways you can test a particular routine helps to define its functionality. Most importantly, however, such unit tests become part of the program. Therefore, testing at that level is not an afterthought, implemented in ad hoc one-off programs.

Testing Frameworks

Junit is a well-known framework for unit tests in a Java programming environment [37]. It has been seminal for a whole range of similar frameworks for all manner, of programming languages. Junit depends on explicit support from several language features that Fortran does not have, such as special annotations and the possibility to programmatically get a list of routine (method) names from a library. Nevertheless, it is quite possible to implement a unit testing framework in Fortran as well.

There is extensive literature on reviewing code and design documents as a means to check that the software is performing as it should and that the implementation is maintainable and understandable. Some development methodologies, such as Extreme Programming [16], prescribe continuous reviews by peers [40]. In all forms, code reviews are about finding defects in the code, but they are not intended to criticize the programmer.

While the procedural (and psychological) aspects of code reviews are widely described, some practical aspects are not. For example, what should you be looking for in the code? It is not enough to check that the code adheres to the programming standard of the project it belongs to. Such a standard may not exist, be incomplete, or be focused on layout, not on questionable constructs that are a liability (see Appendix B).

What do you want to achieve with code reviews? First, you want to verify that the code is doing its job and that the code is of good enough quality:

1. ■ It is readable by others than the author.

2. ■ It is maintainable – adding new functionality or correcting bugs should not amount to hard labor.

3. ■ It is testable – do you understand what goes in and what comes out?

Secondly, the code should be portable. This means it should be possible to build the program and run it on a different operating system, and also with different compilers and versions of that same compiler. Many compilers offer options to check against the language standard.