Let X be a Markov process on the line. Under certain conditions it is possible to find a diffusion process which is an approximation to X in the following sense:

(1) X can be embedded in that is there are stopping times (Tt) such that {Xt, t ≥ 0} and have the same distribution;

(2) for each t, E{Tt} = t.

We call the well-timed diffusion approximation to X, and suggest that it is useful for approximating quantities like the first-exit probabilities and expected first-exit times of X.

We determine the well-timed approximation in several special cases and give an asymptotic approximation for use in cases in which cannot be exactly determined.

This paper deals with asymptotic expansions of the distribution of the kth-largest order statistic Zn–k+1:n for the sample size n. These expansions establish higher-order approximations which hold uniformly over all Borel sets. In the particular case of the distribution of Zn–k+1:n under the uniform distribution and the exponential distribution, the approximating measures are linear combinations of ‘negative’ gamma distributions and linear combinations of extreme-value distributions. These results can be extended to the case of the joint distribution of the k largest order statistics. A numerical comparison to a different asymptotic expansion is given where the normal distribution is the leading term.

For a broad class of stochastic processes, the generalized semi-Markov processes, conditions are known which imply that the steady state distribution of the process, when it exists, depends only on the means, and not the exact shapes, of certain lifetime distributions entering the definition of the process. It is shown in the present paper that this insensitivity extends to certain average and conditional average residence times. Particularly interesting applications can be found in the field of networks of queues.

Daniels (1974) reduced the problem of approximating the distribution of the maximum size of a closed epidemic to that of finding the distribution of max0≦t≦2 {W(t) – N1/2c(t)}, where c is a smooth function with a unique minimum of 0 at t = 1, and he derived an approximation to this distribution which he showed to be accurate to order N–1/4. In this paper, his approximation is shown to be accurate to order N–1/3, and a refined approximation is given which is accurate to order N–1/2 log N. The new approximation is still normal, and its accuracy is similar to that of the original approximation of a discrete process by the Wiener process.

Consider a bundle that has been constructed by placing n fibers in parallel between two clamps. The simplest assumption on fiber load sharing is that all non-failed fibers share the load applied to the bundle equally. Phoenix (1978b) shows that, as n tends to ∞, the time to failure of such a bundle is asymptotically normally distributed under certain assumptions on the stochastic nature of the fibers. In this paper we show that this asymptotic normality is preserved but with changed mean and variance if random slacks are introduced, as caused by random variations in the lengths of the fibers. A theorem on random linear combinations of order statistics, which is given in the appendix, is used to prove the result.

The general framework of a Markov chain in a random environment is presented and the problem of determining extinction probabilities is discussed. An efficient method for determining absorption probabilities and criteria for certain absorption are presented in the case that the environmental process is a two-state Markov chain. These results are then applied to birth and death, queueing and branching chains in random environments.

A GI/G/r(x) store is considered with independently and identically distributed inputs occurring in a renewal process, with a general release rate r(·) depending on the content. The (pseudo) extinction time, or the content, just before inputs is a Markov process which can be represented by a random walk on and below a bent line; this results in an integral equation of the form gn+1(y) = ∫ l(y, w)gn(w) dw with l(y, w) a known conditional density function. An approximating solution is found using Hermite or modified Hermite polynomial expansions resulting in a Gram–Charlier or generalized Gram–Charlier representation, with the coefficients being determined by a matrix equation. Evaluation of the elements of the matrix involves two-dimensional numerical integration for which Gauss–Hermite–Laguerre integration is effective. A number of examples illustrate the quality of the approximating procedure against exact and simulated results.

Using the isomorphism between convex subsets of Euclidean space and continuous functions on the unit sphere we describe the probability measure of the convex hull of a random sample. When the sample is spherically symmetric the asymptotic behavior of this measure is determined. There are three distinct limit measures, each corresponding to one of the classical extreme-value distributions. Several properties of each limit are determined.

In [5] Doeblin considered some classes of finite non-homogeneous Markov chains and gave without proofs several results concerning their asymptotic behaviour. In the present paper we first attempt to make Doeblin's results precise and try to reconstruct his arguments. Subsequently we investigate more general situations, where a state space decomposition is provided by the sets occurring in the representation of the atomic sets of the tail σ-field. We show that Doeblin's notion of an associated chain, as well as considerations regarding the tail σ-field structure of the chain, can be used to solve such cases.

The conventional linear/quadratic/Gaussian assumptions are modified in that minimisation of the expectation of cost G defined by (2) is replaced by minimisation of the criterion function (5). The scalar –θ is a measure of risk-aversion. It is shown that modified versions of certainty equivalence and the separation theorem still hold, that optimal control is still linear Markov, and state estimate generated by a version of the Kalman filter. There are also various new features, remarked upon in Sections 5 and 7. The paper generalises earlier work of Jacobson.

Characteristics of queues with non-stationary input streams are difficult to evaluate, therefore their bounds are of importance. First we define what we understand by the stationary delay and find out the stability conditions of single-server queues with non-stationary inputs. For this purpose we introduce the notion of an ergodically stable sequence of random variables. The theory worked out is applied to single-server queues with stationary doubly stochastic Poisson arrivals. Then the interarrival times do not form a stationary sequence (‘time stationary’ does not imply ‘customer stationary’). We show that the average customer delay in the queue is greater than in a standard M/G/1 queue with the same average input rate and service times. This result is used in examples which show that the assumption of stationarity of the input point process is non-essential.

We derive conditions for the time-uniform continuity of random sequences with respect to variations of governing parameters, and also obtain some estimates of the modulus of continuity. We then apply the results to find conditions of continuity for Markov chains with arbitrary state space, and to construct finite approximations to them.

It is shown that, in a multiserver queue with interarrival and service-time distributions of phase type (PH/PH/c), the waiting-time distribution W(x) has an asymptotically exponential tail, i.e., 1 – W(x) ∽ Ke–ckx. The parameter k is the unique positive number satisfying T*(ck) S*(–k) = 1, where T*(s) and S*(s) are the Laplace–Stieltjes transforms of the interarrival and the service-time distributions. It is also shown that the queue-length distribution has an asymptotically geometric tail with the rate of decay η = T*(ck). The proofs of these results are based on the matrix-geometric form of the state probabilities of the system in the steady state.

The equation for k shows interesting relations between single- and multiserver queues in the rates of decay of the tails of the waiting-time and the queue-length distributions.

The parameters k and η can be easily computed by solving an algebraic equation. The multiplicative constant K is not so easy to compute. In order to obtain its numerical value we have to solve the balance equations or estimate it from simulation.

We extend the optimal strategy results of Kelly and Breiman and extend the class of random variables to which they apply from discrete to arbitrary random variables with expectations. Let Fn be the fortune obtained at the nth time period by using any given strategy and let Fn∗ be the fortune obtained by using the Kelly–Breiman strategy. We show (Theorem 1(i)) that Fn/Fn∗ is a supermartingale with E(Fn/Fn∗) ≤ 1 and, consequently, E(lim Fn/Fn∗) ≤ 1. This establishes one sense in which the Kelly–Breiman strategy is optimal. However, this criterion for ‘optimality’ is blunted by our result (Theorem 1(ii)) that E(Fn/Fn∗) = 1 for many strategies differing from the Kelly–Breiman strategy. This ambiguity is resolved, to some extent, by our result (Theorem 2) that Fn∗/Fn is a submartingale with E(Fn∗/Fn) ≤ 1 and E(lim Fn∗/Fn) ≤ 1; and E(Fn∗/Fn) = 1 if and only if at each time period j, 1 ≤ j ≤ n, the strategies leading to Fn and Fn∗ are ‘the same’.

The problem studied is that of controlling a finite Markov chain so as to maximize the long-run expected reward per unit time. The chain's transition probabilities depend upon an unknown parameter taking values in a subset [a, b] of Rn. A control policy is defined as the probability of selecting a control action for each state of the chain. Derived is a Taylor-like expansion formula for the expected reward in terms of policy variations. Based on that result, a recursive stochastic gradient algorithm is presented for the adaptation of the control policy at consecutive times. The gradient depends on the estimated transition parameter which is also recursively updated using the gradient of the likelihood function. Convergence with probability 1 is proved for the control and estimation algorithms.