Improving Python code

This chapter is concerned with improving Python code and we will illustrate, using short code snippets, various tips that help with speed, memory use and coding clarity. There may be several aspects of a program that we seek to improve, but we can’t necessarily expect to improve all of them all of the time. Often optimisation of a Python program is about compromise; you may make a program run faster at the expense of using more memory. Clarity is an especially important aspect that we will be mindful of when making suggestions, and in general we recommend making code more easily understood over mild improvements in performance. Finding and correcting errors in code can take a long time, sometimes longer than the program took to write in the first place, so keeping the code easy to understand is especially important.

A basic programming approach that the authors often follow, and which may be helpful for others, is a three-point plan:

1. Firstly, make the code work: an inelegant program is better than one that doesn’t work.

2. Next do it properly: with a working reference, you can take a step back and criticise your approach.

3. Then make it better: only once your program is working, and the general approach won’t change, is it worth optimising.

Biological images

Often in biology and medicine the data people use comes in the form of an image. This could be as simple as a photograph of some cells or an image that has been constructed from other data, e.g. from an MRI scan. The images that we will be discussing in this chapter, whatever their source, will be pixmap images, also known as raster images. They will be constructed as rectangular arrays of colour or grey values, the smallest square element of which we refer to as a pixel. We will not be considering the vector graphics approach to making pictures, where the data is described in terms of lines and shape outlines. Here we will concentrate on pixel arrays, the kind of image data that comes from our digital cameras and various scientific instruments.

We will deal with pixmap images in a general, slightly mathematical way. It will not matter what the image actually represents for the most part, although we will endeavour to give examples with a biological flavour. Not so long ago images would largely be acquired by using photographic film, but now the digital camera is ubiquitous, and without the need to buy expensive film a scientist can capture as many images as time and storage capacity allow. Thus the examples presented here will often have an emphasis towards automation, and if you need to write programs dealing with biological data this will allow you to construct efficient analytical pipelines.

Data models

This chapter delves more deeply into the topic of creating custom Python objects using class definitions. Given that we have discussed the basics of object-oriented programming in Chapter 7, we now move on to illustrate how such mechanisms can be used in a practical, scientific sense. If you are interested in only a light introduction to Python, you might consider skipping this chapter on a first reading. However, the objects we discuss here will underpin many of the examples given later on in this book, in Chapters 15 and 20, so you may like to look back to see how such things are constructed.

In the previous chapter we saw how to introduce our own types of data object into Python, using classes. Here we move on to look at how to use a number of different, but connected, classes to construct what is often known as a data model. A data model is an abstract description of concepts that can be used to build a computational version of some topic or real-world situation that you are interested in. Essentially, you examine the kind of information you wish to describe and divide it up into conceptual parcels. Each of these will become one kind of computer object (a class with attributes, functions and links to other classes), which then allows you to create a synthetic analogue of the thing you are interested in. Virtually all programs, irrespective of size, rely on some kind of underlying model to organise data, although this may not use object-oriented programming and is often not formalised in any way. No data model can be expected to be a perfect computer representation of what it describes, but the idea is to make it good enough to serve a useful purpose, by having some of the properties of the things being modelled.

Running things faster

This chapter is all about how to make Python programs run faster. We will discuss optimising existing routines so that they take a shorter amount of time to run, above and beyond the simple Python tips and tricks discussed earlier. Initially parallel computing, where a job is split into parts and run concurrently on separate processors (or processing cores), is discussed in a basic way. For this we use modules that are available from Python 2.6 and above, which allow programs to take account of multiple processing cores present in a single computer. For the remainder of the chapter we will deal with improving the performance of a single processing job.

At the end some timing results will be given so that the reader can see how much was gained for the effort. For mathematical routines involving lots of loops it is not uncommon to get speed improvements of better than tenfold. The fine details about the logic and underlying algorithms of the examples used here will not be described; an example will be taken from earlier in the book where such things are described fully. Also, which particular example we have chosen is not especially important, other than the fact that it is a computationally intensive one that takes a noticeable time to run. It should be noted that this chapter comes with a ‘health warning’ for novice programmers, because the mainstay of the optimisation will be to move away from Python. Some of the focus will be on the low-level compiled language C, although it will be used in a way to provide a module that can still be used directly inside Python programs. The details of the C language, and how to compile it, will not be discussed and to actually learn to program in C we recommend further reading. Nonetheless, if you have no experience with C we hope that we can provide a basic appreciation of how it can help. We also consider Cython, a C-like extension to Python, which has made it possible to benefit from the speed of C without having to necessarily deal with all the complexities of C. This is particularly powerful in combination with using NumPy arrays.

Using Python for mathematics

Given that Python is an interpreted programming language, rather than a fast compiled language, many people do not consider it for writing programs that involve extensive numerical work. While Python programs are certainly slower to execute than the equivalent written in something like C or FORTRAN, mathematical functionality certainly exists in Python and has the inherent advantages of the language; it is easy for people to use and conveniently links to other helpful data structures. Of course speed of calculation may not be so important, for a scientific investigation it may not matter if something takes 1 second or 0.1 second to run. Fortunately, computers get faster and the Python interpreter becomes improved, so you can do quite a bit of numerical work without concern. However, if calculation speed really is important in a given situation then there are a few things you can do to make things faster while still keeping the convenience of Python. For example, you can write code in C, a very efficient numerical language, and use it from within Python (this is called a C extension), effectively extending the vocabulary of the interpreted language with speedy subroutines. More recently the language Cython has helped make C extensions very easy to write. Cython is a Python-like language, and virtually all Python programs can be interpreted by it, without alteration, but the language ultimately generates C code that can be compiled. Cython can be used to call fast library code written in pure C, and can incorporate a mixture of Python and C data structures in the same code; although less flexible, the C data structures are very efficient. Writing C extensions and Cython modules is discussed in Chapter 27.

Python includes standard arithmetic operations as part of the core functionality: add, multiply etc. There is an additional module, math, which always comes packaged with Python and which provides further numerical functionality: logarithms, trigonometry etc. For numerical calculations that are not especially intensive, the core functionality and the math module will often suffice. There has been a history of trying to provide modules for quick numerical algorithms in Python. The first attempt, begun in 1995, was called Numeric, and the second attempt was called Numarray. These two are now deemed to be obsolete, but the third attempt, begun in 2005, is called NumPy (http://numpy.scipy.org/), incorporates elements from the earlier attempts and will hopefully last longer.

Programming principles

The Python language can be viewed as a formalised system of understanding instructions (represented by letters, numbers and other funny characters) and acting upon those directions. Quite naturally, you have to put something in to get something out, and what you are going to be passing to Python is a series of commands. Python is itself a computer program, which is designed to interpret commands that are written in the Python language, and then act by executing what these instructions direct. A programmer will sometimes refer to such commands collectively as ‘code’.

Interpreting commands

So, to our first practical point; to get the Python interpreter to do something we will give it some commands in the form of a specially created piece of text. It is possible to give Python a series of commands one at a time, as we slowly type something into our computer. However, while giving Python instructions line by line is useful if you want to test out something small, like the examples in this chapter, for the most part this method of issuing commands is impractical. What we usually do instead is create all of the lines of text representing all the instructions, written as commands in the Python language, and store the whole lot in a file. We can then activate the Python interpreter program so that it reads all of the text from the file and acts on all of the commands issued within. A series of commands that we store together in such a way, and which do a specific job, can be considered as a computer program. If you would like to try any of the examples given in the book the next chapter will tell you how to actually get started. The initial intention, however, is mostly to give you a flavour of Python and introduce a few key principles.

An introduction to graphical user interfaces

At some stage when writing your own programs there may come a time when you want others to be able to use what you have created without them necessarily having to know anything about programming or Python. Should this happen the next step is to consider writing a more friendly interface to the program. Once upon a time in computing everything was text-based and the user had to type commands to get things to work. Fortunately things have moved on and we are now usually presented with graphics and a pointing device, either a mouse or a touch screen, and the user can interact with graphical objects like menus and buttons.

When building a graphical user interface (GUI) the programmer must be mindful of various factors, which are sometimes antagonistic, forcing us to make compromises. For example, the designer has to strike a balance between on the one hand giving lots of functionality and on the other hand keeping things simple for novices and intuitive to use. In this chapter we will aim to give some general advice about the programming, but we leave you to make the tough choices. We wish to be clear that this chapter deals with making graphical interfaces that run on the users’ local computer. We will not venture into the world of Internet-based applications, although these are becoming increasingly important, and the Pyjamas library, which is available for Python programmers, works in a remarkably similar way to the graphical libraries discussed here.

Multiplexed experiments

In many areas of biological and medical science, as new techniques and machinery are developed there is a tendency to record ever increasing amounts of data. A notable example of this is comes with ‘next-generation’ DNA sequencing, which we discuss further in Chapter 17. In general though, with high-throughput methods the idea is to perform many small experiments, of the same design, in parallel. When we simultaneously detect the outcome of many assays the procedure can be described as being multiplexed. This not only has speed advantages but can also reduce costs and improve consistency between experiments. And naturally, to handle large numbers of experimental assays it is important to use computers for the processing and analysis of data.

A multitude of modern techniques involve parallel experiments, including the detection of potential drug compounds, RNA molecules, antibodies and protein crystals, to name only a few. However, in this chapter we do not have space to cover the informatics of lots of specific techniques, so instead we cover general themes, such as data organisation, normalisation and comparison. Also, all of the examples will be based on the notion of the experimental data being arranged as a rectangular array, which in turn is often a consequence of the physical manner in which the assays were performed and detected, on some form of regular grid.

Creating classes

For simple tasks involving short programs, you can survive perfectly well with the standard Python data types for holding information, such as lists and dictionaries. However, for more complicated tasks involving long programs, this often becomes unwieldy. There are various ways to deal with this issue, but one of the most fruitful is the ability to define your own data types: objects built to your own specification, organised in the way that is convenient to you. Modern computer languages do this via the introduction of bespoke object definitions that are known as classes and this kind of thinking is generally termed object-oriented programming.

When creating your own custom data types, the class is the definition of a particular kind of object in terms of its component features and how it is constructed or implemented in code. The term object, however, refers to a specific instance, or occurrence, of the thing which has been made according to the class definition. The making of an object of a given class is what is usually termed instantiation. A convenient analogy is to think of the blueprint for a house being like a class, but the actual, solid house being the object. Also, given a single blueprint one may build many instances of different house objects, all to the same design. It is quite common to use the words ‘class’ and ‘object’ interchangeably, even in the same context, although they mean different things, and it is important to understand the difference. As it happens, everything that is brought into existence in Python is an object, so even integer and floating point numbers are objects, although most of the time you can work without noticing that.

The basics of probability theory

The theory of probability was based on the observation of random physical events, most notably for games of chance. And naturally, calculating accurate probabilities became especially important for people when money was wagered on the outcome. Probability is a way of ascribing numerical values to the possible outcomes to help us understand a random process more fully. This enables us to ask questions like how much more often one event occurs compared to another, but because of the random nature of what we are studying we can never say what the outcome will definitely be. Rather we tend to think of the process in terms of what the long-term proportions of different outcomes are, if the random experiment were repeated a very large number of times, or perhaps if money is involved what a wager on a particular outcome is worth.

Turning to biological systems, some things in living organisms occur as a result of random processes, like the segregation of a parent’s chromosomes among their children or base-pair changes in DNA (such as a result of replication errors or ionising radiation), though, under most circumstances we don’t get to see the actual random event. For the most part we just view the outcomes, sometimes billions of years later in the case of DNA sequence changes. Of course a DNA sequence isn’t actually random, given that it exists to contain biologically meaningful information representing genes and gene control elements etc. which have been selected for their function during evolution, even if the initial mutations were random. Nonetheless for a sufficiently large and unbiased selection of DNA we can treat the sequence as if it were random in order to ask various questions. For example, how often do I find the sub-sequence AAGCTT in a megabase-long region of DNA?

Many years ago we started programming in Python because we were working on a large computational biology project. In those days choosing Python was not nearly as common as it is today. Nonetheless things worked out well, and as our expertise grew it seemed only natural that we should run some elementary Python courses for the School of Biology at the University of Cambridge, where we were employed. The basis for those courses is what turned into the initial idea for this book. While there were many books about getting started with Python and some that were tailored to bioinformatics, we felt that there was still some room for what we wanted to put across. We began with the idea that we could write some chapters in relatively straightforward English that were aimed at biologists, who might be complete novices at programming, and have other sections that are useful to a more experienced programmer. Also, given that we didn’t consider ourselves to be typical bioinformaticians, we were thinking more broadly than just sequence-based informatics, though naturally such things would be included. We felt that although we couldn’t anticipate all the requirements of a biological programmer there were nonetheless a number of key concepts and techniques which we could try to explain. The end result is hopefully a toolkit of ideas and examples which can be applied by biologists in a variety of situations.

Acknowledgements

We extend our sincere thanks to a group of intrepid volunteers who have been invaluable in the proof-reading and testing of this book: Olga Tkachenko, Magnus Lundborg, Neil Rzechorzek, Rasmus Fogh, Simon Fraser and Tom Drury.

Controlling command execution

On the whole, a program will normally run by executing the stated commands, one after the other in sequential order, like reading the lines of a book. Frequently, however, you will need the program to deviate from this, to jump to a line that is not the next line. There are three main ways of diverting from the line-by-line paradigm. The first way is through the use of functions (subroutines), where the program’s execution jumps from a particular line of code to an entirely different spot, even in a different file or module, to do a task before (usually) jumping back again. You can even jump to a function’s subroutine from inside another function, and do this repeatedly, so that there is a stack of jumps between the current line and the first jump point. Given their importance, functions will be discussed in a dedicated chapter that follows this one.

The second way of jumping between program lines is through the use of conditional statements. Here you can check if some statement or expression is true, and if it is then you continue on with the following block of code, otherwise you might skip it or execute a different bit of code. The third way is by performing repetitive loops through the same lines of code, where each time through the loop different values may be used for the variables. Usually such loops are either done a specific number of times or until something in particular happens. When you have a collection of items, like a list, a loop can be used to consider all of the items in turn; the loop iterates over the items of the collection. A significant number of people who are new to programming find this the hardest idea to get to grips with, although Python’s syntax makes it about as easy as it can be.

Python programming for biology

One of the main aims of this book is to empower the average researcher in the life sciences, who may have a pertinent scientific question that can be readily answered by computational techniques, but who doesn’t have much, if any, experience with programming. For many in this position, the task of writing a program in a computer language is a bottleneck, if not an impassable barrier. Often, the task is daunting and seems to require a significant investment of time. The task is also subject to the barriers presented by a vocabulary filled with jargon and a seemingly steep learning curve for those people who were not trained in computing or have no inclination to become computer specialists. With this in mind for the novice programmer, one ought to start with the language that is the easiest to get to grips with, and at the time of writing we believe that that language is Python. This is not to say that we have made a compromise by choosing a language that is easy to learn but which is not powerful or fully featured. Python is certainly a very rich and capable way of programming, even for very large projects; otherwise we authors wouldn’t be using it for our own scientific work.

A second main aim of this book is to use Python as a means to illustrate some of what is going on within biological computing. We hope our explanations will show you the scientific context of why something is done with computers, even if you are a newcomer to biology or medical sciences. Even where a popular biological program is not written in Python, or if you are a programmer who has good reason for using another language, we can still use Python as a way of illustrating the major principles of programming for biology. We feel that many of the most useful biological programs are based on combinations of simple principles that almost anyone can understand. By trying to separate the core concepts from the obfuscation and special cases, we aim to provide an overview of techniques and strategies that you can use as a resource in your own research. Virtually all of the examples in this book are working code that can be run and are based on real problems or programs within biological computing. The examples can then be adapted, altered and combined to enable you to program whatever you need.

A basic introduction to sequence variation

Naturally the genetic codes of different kinds of organism differ to support their different construction, habits and biological requirements. Also, genetic codes vary between individuals of the same species, despite the large degree of similarity that binds them as a species. It is such variation within species that provides the opportunity for offspring to differ from their parents and potentially gain an improvement or specialisation, which in time may give rise to a new species. In a more modern context, sequence variations have a vital role in our understanding of genetic diseases and are becoming increasingly important for the development of pharmaceuticals, where the effectiveness and side effects of drugs may vary significantly according to the genotype of a person.

The variety of species and individuals is all down to the variety of genome sequences, but in order to discover as much as we can about the consequences of and reasons for this variation we should understand something about the mechanism by which sequences can change. In this chapter we will not go in to immense detail about the underlying mechanics, but simply cover the main principles and, importantly as far as bioinformatics is concerned, describe what kinds of change are detectable in the biological sequences. It should be remembered that it is only changes that are passed on to offspring which will influence evolution directly, but other variations may be important, for example, in the study of cancer-causing mutations.