4 - Augmented Statistical Mechanics Formulation

Summary

The generalization performance of some of the learning rules introduced in the previous chapter could be characterized either by using simple arguments from statistics as in the case of the Hebb rule, or by exploiting our results on Gibbs learning obtained in chapter 2 as in the case of the Bayes rule. Neither of these attempts is successful, however, in determining the generalization error of the remaining learning rules.

In this chapter we will introduce several modifications of the central statistical mechanics method introduced in chapter 2 which will allow us to analyse the generalization behaviour of these remaining rules. The main observation is that all these learning rules can be interpreted as prescriptions to minimize appropriately chosen cost functions. Generalizing the concept of Gibbs learning to non-zero training error will pave the way to studying such minimization problems in a unified fashion.

Before embarking on these general considerations, however, we will discuss in the first section of this chapter how learning rules aiming at maximal stabilities are most conveniently analysed.

The main results of this chapter concerning the generalization error of the various rules are summarized in fig. 4.3 and table 4.1.

Maximal stabilities

A minor extension of the statistical mechanics formalism introduced in chapter 2 is sufficient to analyse the generalization performance of the adatron and the pseudo-inverse rule. The common feature of these two rules is that they search for couplings with maximal stabilities, formalized by the maximization of the stability parameter K.