A huge amount of physiological research shows the importance of the role of the cerebellum in motor control (Ito, 1984; and many others). It is natural for physiologists to try to understand what the cerebellum function is and when and which corrections or other modifications of cerebral cortical motor programs are introduced by the cerebellum. It seems to us that in recent decades the most interesting hypotheses on cerebellar performance have been proposed by the late David Marr in ‘A theory of cerebellar cortex’ (Marr, 1969). According to Marr, the main operation of the cerebellar circuitry is the switching on of proper motor commands by the current sensory input and the automatic adaptive acquisition of such cerebellar network capability. The location of the cerebellum – in the crest of almost all the ascending and descending nervous tracts – is definitely strategic for such a function. The unique combination of tens of thousands of granular cells (GrCs) and one climbing fibre (CF) at one Purkinje cell seems to be crucial for it.

The kernel of Marr's theory is composed by the postulates of the GrC–PC synaptic modification due to simultaneous excitation of the climbing fibre (CF) and parallel fibres (PF). In other words, Marr supposed that the Purkinje cell memorizes the afferent conditions in which it ought to be active.

Introduction

Rhythmic behaviour is characteristic for the nervous system at different hierarchical levels. Periodic temporal patterns can be generated both by endogenous pacemaker neurons and by multicellular neural networks. At single-cell level it was demonstrated, both experimentally and theoretically, that periodic membrane potentials could bifurcate to more complex oscillatory behaviour (ultimately identified by chaos) in response to drug treatment (Holden, Winlow & Haydon, 1982; Chay, 1984). Even the alteration of periodically synchronized oscillation and chaotic behaviour has been found in periodically forced oscillators of squid giant axons (Aihara, Matsumoto & Ichikawa, 1985).

The appearance of quasi-periodicity and chaos has been associated with abnormal neural phenomena not only at single neural level but as well at macroscopic scale connecting chaotic EEG dynamics to epileptic seizure (Babloyantz, Salazar & Nicholis, 1985). At intermediate level, chaotic behaviour was found in a model of the central dopaminergic neuronal system, and was associated with schizophrenics (King, Barchas & Huberman, 1984).

‘Normal’ and ‘abnormal’ dynamic behaviour, also at intermediate, namely synaptic level, has recently been investigated (Érdi & Barna, 1986; Éedi & Barna, 1987). Preliminary numerical calculations suggested that the regular periodic operation of synaptic level rhythmic generator of cholinergic system requires a fine-tuned neurochemical control system. Even mild impairment of the metabolism might imply ‘abnormal’ dynamic synaptic activity.

Memory disorders associated with Alzheimer's disease, partially due to disturbance of the control system of acetylcholine (ACh) synthesis, can be accompanied by change of dynamic patterns of firing frequency (Wurtman, Hefti & Melamed, 1981).

Introduction

The belief that complex macroscopic phenomena of everyday experience are consequences of cooperative effects and large-scale correlations among enormous numbers of primitive microscopic objects subject to short-range interactions, is the starting point of all mathematical models on which our interpretation of nature is based. The models used are basically of two types: continuous and discrete models.

The first have been until now the most current models of natural systems; their mathematical formulation in terms of differential equations allows analytic approaches that permit exact or approximate solutions. The power of these models can be appreciated if one thinks that complex macroscopic phenomena, such as phase transitions, approach to equilibrium and so on, can be explained in terms of them when infinite (thermodynamic) limits are taken.

More recently, the great development of numeric computation has shown that discrete models can also be good candidates to explain complex phenomena, especially those connected with irreversibility, such as chaos, evolution of macroscopic systems from disordered to more ordered states and, in general, self-organizing systems. As a consequence, the interest in discrete models has vastly increased. Among these, cellular automata, C.A. for short, have received particular attention; we recapitulate their definition:

A discrete lattice of sites, the situation of which is described at time t by integers whose values depend on those of the sites at the previous time t - τ (τ is a fixed finite time delay).

Introduction

Neural networks of spin glass type reveal remarkable properties of a content-addressable memory (Hopfield, 1982; Amit et al, 1985; Kinzel, 1985a). They are able to retrieve the full information of a learned pattern from an initial state which contains only partial information. Recently much effort has been devoted to the modeling of networks based on Hebb's learning rule (Cooper et al., 1979). These networks are the Hopfield model and its modifications. All have in common a local learning rule which allows the storage of orthogonal patterns without errors. The learning rule is local if the change of the synaptic coefficient depends only on the states of the two interconnected neurons and possibly on the local field of the postsynaptic one. This property seems to be essential from a biological point of view. However, the storing capability of these networks is strongly limited by the fact that they are not able to store correlated patterns without errors (Kinzel, 1985b).

On the other hand a storing procedure for correlated patterns is available (Personnaz et al, 1985; Kanter & Sompolinsky, 1986). But it involves matrix inversions which are not equivalent to a local learning mechanism. It is the purpose of this paper to present a new local learning rule for neural networks which are able to store both correlated and uncorrelated patterns. Moreover, this learning rule enables the network to fulfil two further important properties of natural networks: the learning process does not reverse the signs of the synaptic coefficients and leads to a network with unsymmetric bonds even if it starts from a symmetric one.

Introduction

The mechanisms of the complex functions attributed mostly to the cerebral cortex are hidden in the collective behaviour of a vast neural network that cannot practically be described in detail or in general. Cyclic modes of activity which emerge spontaneously in the dynamics of neural networks may underly possible mechanisms of short-term memory and associative thinking. The transitions from seemingly random activity patterns to cyclic activity have been examined in isolated networks with pseudorandomly chosen synapses and in networks with very simple architectures.

The basic computer model (Clark, Rafelski & Winston, 1985) envisions a collection of neurons, linked by a network of axons and dendrites that synapse onto one another. The synaptic interactions are modeled by a connection matrix V. The net algebraic strength of the connections from neuron j to neuron i, represented by the matrix element Vij can be positive (excitatory), negative (inhibitory) or zero (no connection). In the present study, the Vij were chosen randomly, but in accord with certain specified gross network parameters, viz.

N = net size = number of neurons in net,

m = connection density = probability that a given j → i link exists,

h = fraction of inhibitory neurons.

No more than one connection (‘synapse’) was allowed from any source neuron j to a given target neuron i.

The neurons update their states synchronously, corresponding to the assumption of a universal time delay δ for direct signal transmission.

The retina

Mathematical model of amphibian retina

The retina is composed of a variety of cell types including the photoreceptors, horizontal cells, bipolar cells, amacrine cells, and retina ganglion cells (for a review see Grüsser & Grüsser-Cornehls, 1976). Only the retina ganglion cells (RGCs) send axons to the brain of the animals. Therefore any visual information the brain may rely on is mediated by cells of this type.

According to their response properties the RGCs are usually divided into four classes, which here for simplicity are called R1, R2, R3, and R4. We have restricted our attention to the classes R2 and R3 because these form the majority (about 93%) of cells projecting to the tectum opticum, which is that area in the brain where recognition of prey objects is supposed to be centered.

The recognition process starts in the retina. The overall operation of the ganglion cells (R2, R3) and their precursors (photoreceptors, etc.) on some arbitrary visual scene can be decomposed into the following more primitive operational components:

(1) Let x(s, t) be any distribution of light in the visual field, where x denotes light intensity (we do not consider colored scenes), s = (s1, s2) some point in the visual field, and t is time.

(2) These ganglion cells do not respond to stationary, but only to transitory, illumination.

Introduction

In dealing with the problem of trying to describe mathematically neural tissue populations, known as neural nets, two approaches can be taken, the global and the microscopic. The global approach gives a phenomenological description of neural tissue populations. The microscopic, derived through appropriate simplifications, gives properties of the net from the properties of its constituents, the neurons, the connections and the synapses. While phenomenological theories appear to be easier to build and, overall, yield more results that are in agreement with experiments, it is impossible to build a realistic model of the brain without knowing the detailed functioning, interactions and interrelations of its constituents.

For the construction of an actual neural net machine, theories based on the properties of the fundamental constituents of the net will be more applicable to discovering the laws that govern the secrets of biological information processing. For instance, it is more relevant to simulate the activities of the brain from the underlying fundamental laws of nature. Examples taken from physics itself clarifies this point.

From a philosophical point of view it is more appealing to derive the laws of thermodynamics from the statistical behaviour of the particles that make up the system than to introduce thermodynamics as an independent branch of physics.

Analogously, it is more significant to derive the properties of superconductivity from the actual properties of the electrons and lattice than from phenomenological reasoning.

Introduction

Many features of the functional architecture of the mammalian visual system have been experimentally identified during the past 25 years. Among the most striking of these features is the presence of layers of orientation-selective cells – cells whose response to an edge or bar in the appropriate portion of visual field is sensitive to the local orientation of the input. These cells are organized in bands or ‘columns’ of cells of the same or similar orientation preference. The preferred orientation varies roughly monotonically, but with frequent breaks and reversals, as one traverses the cell layer. Orientation-selective cells, organized in this fashion, are found in cat, monkey, and other mammalian systems. In macaque monkey, they are present at birth.

I have found that several salient features of mammalian visual system architecture – including orientation-selective cells and columns – emerge in a multilayered network of cells whose connections develop, one layer at a time, according to a synaptic modification rule of Hebb type. The theoretical base is biologically plausible, none of the assumptions is specific to visual processing, no orientation preferences are specified to the system at any stage, and the features emerge even in the absence of environmental input.

The development of this system is discussed in detail, and references to experimental work are provided, in a series of three papers (Linsker, 1986a,b,c).

Introduction

The fundamental aim of simulation of neural nets is a better understanding of the functioning of the nervous system. Because of the complexities, simplifying assumptions have to be introduced from the beginning. In many frequently used models these assumptions are:

Each model neuron can be in one of few discrete states (e.g. ‘on’, ‘off’, ‘refractory’).

The totality of interactions between neurons is treated summarily by specifying few parameters, often just one ‘synaptic strength’, which is determined randomly, or by a deterministic algorithm, or by a combination of both.

The information processing by a neuron consists of comparing the value of a certain parameter, which is determined by incoming signals from other neurons, with some threshold. Essentially, when the value of this parameter exceeds the threshold, the neuron transits to another one of its possible states. A typical example for this parameter is the membrane potential at the axon hillock, which is determined by summing the action potentials impinging on this neuron during a certain time and whose value determines whether the neuron ‘fires’ or not.

The hypothesis that individual neurons can be described in a simple way makes the simulation of networks containing many model neurons possible. On the other hand, there are important parts of nervous systems, where model neurons simplified to the extent described above have little in common with reality.

Motivation

A neuron's electrophysiological response to a disturbance is within the millisecond range and consists of action potentials which are produced when the threshold potential is being crossed. This response can be measured with microelectrodes and related to the stimulus. With neuroanatomical responses, this is a completely different task. Morphogenetic changes, such as reactive synaptogenesis or degeneration, take hours to days, so they cannot be directly observed by microelectrodes or the like. Besides, it would be very hard to relate the measured functional changes of electrical signals to changes in the synaptical ‘hardware’. Therefore, animals have to be taken at different stages of time and the development of the central nervous system (with or without disturbances) has to be concluded from changes of their individual morphology. Moreover, to get reliable data, several animals have to be sacrificed at each time step. Sometimes, even that may not be enough, as the following example may show.

Measuring the development of the number of synapses in rat cortex from postnatal day 2 to adulthood (Balcar, Dammasch & Wolff, 1983), we expected to fit the data with a sigmoid curve corresponding to logistic growth, but the material appeared to be systematically disturbed. A population-kinetic model (Wagner & Wolff, subm.) developed at that time, could explain the disturbance as a temporary overshoot of free synaptic offers that were not distinguishable from bound synaptic elements.

In deciding on a chapter for this book I had to choose between my interests in bringing ideas from biology to computational hardware, and in trying to bring computational ideas towards the biological wetware. I chose the direction of biological wetware, but it is instructive to begin with a few words about VLSI ‘neural’ networks.

There have been sizeable associative memories built out of the VLSI kinds of hardware. A first effort succeeded in building a 22 neuron circuit (1). A 54-neuron circuit with its 3000 connections has been made at Bell Laboratories (2) and a 512-neuron circuit with its 256000 intersections has been fabricated, but not yet made to work (3). An interesting and significant feature which emerges from looking at these chips is the degree to which, although the idea of an associative memory is very simple, a large fraction of the devices and area on the chip are doing things which are essential to the overall function of the chip, but which seem peripheral to the idea of associative memory. I will describe a similar facet in the biological case, in going from the abstract idea of an associative task toward a complete neural system. Much paraphernalia must be added and many changes made in order that an elementary associative memory can perform a biological task.

Introduction

The detection of image regions and their borders is one of the basic requirements for further (object domain-) image processing in a generalpurpose technical pattern recognition system and, very likely, also in the visual system. It is a pre-requisite for object separation (figure-ground discrimination, and separation of adjoining and intersecting objects), which in turn is necessary for the generation of invariances for object recognition (Reitboeck & Altmann, 1984).

Texture is a powerful feature for region definition. Objects and background usually have different textures; camouflage works by breaking this rule. For texture characterization, Fourier (power) spectra are frequently used in computer pattern recognition. Although the signal transfer properties of visual channels can be described in the spatial (and temporal) frequency domain, there has been no conclusive evidence that pattern processing in the primary visual areas would be in terms of local Fourier spectra.

In the following we propose a model for texture characterization in the visual system, based on region labeling in the time domain via correlated neural events. The model is consistent with several basic operational principles of the visual system, and its texture separation capacity is in very good agreement with the pre-attentive texture separation of humans.

Texture region definition via temporal correlations

When we look at a scene, we can literally generate a ‘matched filter’ and use it to direct our attention to a specific object region.

Is there enough motivation for a solid-state physics approach to the brain?

One of the salient features of the brain networks is that anatomical sections of a few millimetres width, taken from different parts of the cortex, look roughly similar by their texture. This observation might motivate a theoretical approach in which principles of solid-state physics are applied to the analysis of the collective states of neural networks. Such a step, however, should be made with extreme care. I am first trying to point out a few characteristics of the neural tissue which are similar to those of, or distinguish it from non-living solids.

Similarities. There is a dense feedback connectivity between neighbouring neural units, which corresponds to interaction forces between atoms or molecules in solids.

Generation and propagation of brain waves seem to support a continuous-medium view of the tissue.

Differences. In addition to local, random feedback there exist plenty of directed connectivities, ‘projections’, between different neural areas. As a whole, the brain is a complex self-controlling system in which global feedback control actions often override the local ‘collective’ effects.

Although the geometric structure of the neural tissue looks uniform, there exist plenty of specific biochemical and physiological differences between cells and connections. For instance, there exist some 40 different types of neural connection, distinguished by the particular chemical transmitter substances involved in signal transmission, and these chemicals vary from one part of the brain to another.

Introduction

It is probably fair to say that we have not, to this day, formed a clear picture of the learning process; neither have we been able to elicit from artificial intelligence machines a sort of behavior which could possibly compare in flexibility and performance with that exhibited by human or even animal subjects.

Leaving aside the issue of what actually happens in a learning brain, research on the question of how to generate ‘intelligent’ behavior has oscillated between two poles. The first, which today predominates in artificial intelligence circles (Nilsson, 1980), takes it for granted that solving a particular problem entails repeated application, to a data set representing the starting condition, of some operations chosen in a predefined set; the order of application may be either arbitrary or determined heuristically. The task is completed when the data set is found to be in a ‘goal’ state. This approach can be said to ascribe to the system, ‘from birth’, the capabilities required for a successful solution. The second approach, quite popular in its early version (Samuel, 1959), has been favored recently by physicists (Hopfield, 1982; Hogg & Huberman, 1985), and rests on the idea that ‘learning machines’ should be endowed, not with specific capabilities, but with some general architecture, and a set of rules, which are used to modify the machines' internal states in such a way that progressively better performance is obtained upon presentation of successive sample tasks.

Introduction

Rhythmic oscillation is a fundamental component which can be found in the various kinds of nervous systems (Friesen & Stent, 1977; Thompson, 1982). In a neural network with a ring-structured set of synaptic connections, a set of oscillations with different phases can be generated (Morishita & Yajima, 1972; Stein et al., 1974), and the occurrence of such rhythmic oscillation is also confirmed in different types of neural networks (Matsuoka, 1985). Since, however, various additional connections can cause a disturbance which easily extinguishes the rhythmic oscillation in the neural network, some function for maintaining the rhythmic oscillation should be expected to exist in the synapses if such signals play an important role in the nervous system.

A new synaptic modification algorithm is proposed which employs the average impulse density (AID) and the average membrane potential (AMP); examination of the effect of synaptic modification on rhythmic oscillation has been attempted (Tsutsumi & Matsumoto, 1984a). Simulation demonstrated some cases in which rhythmic oscillation reappears, by applying the algorithm to the disturbed ring neural network where the rhythmic oscillation was previously extinguished.

If that is the case, how can such oscillation derived from the neural network with feedback inhibition be processed in the following neural network with, for example, the feedforward system? Here we take, as an instance, the cerebellar circuitry including both feedback and feedforward systems, and discuss the relationship between synaptic modification and rhythmic oscillation in the neural network.

Introduction

Modeling the brain requires some a priori description of its functional computing elements. The sophistication of this description depends on the goal of the model, but it is clear that describing single neurons as simple ‘leaky-integrators’ with a non-linear threshold has limited utility in exploring the capability of the neural net for processing information. In fact, cortical pyramidal neurons are functionally quite complicated, both in terms of how their complex geometry determines their electrotonic structure, and in terms of the active currents that modulate the linear response of these cells.

Over the past several years investigators have uncovered a plethora of currents in one type of cortical pyramidal neuron, the hippocampal pyramidal cell. The currents, which presumably are mediated by distinct ion channels, include the classical fast sodium current (INa), a persistent sodium current (INaP) (French & Gage, 1985), a delayed-rectifier potassium current (IDR) (Segal & Barker, 1984), a calcium current (ICa) (Halliwell, 1983), a slow calcium current (ICaS) (Johnston et al., 1980), a fast transient calcium-mediated potassium current (Ic) (Brown & Griffith, 1983), an after-hyperpolarization calcium-mediated potassium current (IAHP) (Lancaster & Adams, 1986), a muscarine-inhibited potassium current (IM) (Halliwell & Adams, 1982), a transient potassium current (IA) (Gustafsson et al., 1982), a chloride current (ICI(V)) (Madison et al., 1986), and a possibly mixed carrier anomalous rectifier current (IQ) (Halliwell & Adams, 1982). Each current has a unique time-and voltage-dependence, and some currents are sensitive to constituents in the extra-or intra-cellular environment, for example free Ca++ or muscarinic agonists.