This work concerns Markov decision chains with finite state spaces and compact action sets. The performance index is the long-run risk-sensitive average cost criterion, and it is assumed that, under each stationary policy, the state space is a communicating class and that the cost function and the transition law depend continuously on the action. These latter data are not directly available to the decision-maker, but convergent approximations are known or are more easily computed. In this context, the nonstationary value iteration algorithm is used to approximate the solution of the optimality equation, and to obtain a nearly optimal stationary policy.

Consider n cells, of which some are target cells, and suppose that each cell has a weight. The cells are killed in a sequential manner, with each currently live cell being the next one killed with a probability proportional to its weight. We study the distribution of the number of cells that are alive at the moment when all the target cells have been killed.

We consider a stochastic model for the spread of a susceptible–infective–removed (SIR) epidemic among a closed, finite population, in which there are two types of severity of infectious individuals, namely mild and severe. The type of severity depends on the amount of infectious exposure an individual receives, in that infectives are always initially mild but may become severe if additionally exposed. Large-population properties of the model are derived. In particular, a coupling argument is used to provide a rigorous branching process approximation to the early stages of an epidemic, and an embedding argument is used to derive a strong law and an associated central limit theorem for the final outcome of an epidemic in the event of a major outbreak. The basic reproduction number, which determines whether or not a major outbreak can occur given few initial infectives, depends only on parameters of the mild infectious state, whereas the final outcome in the event of a major outbreak depends also on parameters of the severe state. Moreover, the limiting final size proportions need not even be continuous in the model parameters.

The purpose of this paper is to determine the exact distribution of the final size of an epidemic for a wide class of models of susceptible–infective–removed type. First, a nonstationary version of the classical Reed–Frost model is constructed that allows us to incorporate, in particular, random levels of resistance to infection in the susceptibles. Then, a randomized version of this nonstationary model is considered in order to take into account random levels of infectiousness in the infectives. It is shown that, in both cases, the distribution of the final number of infected individuals can be obtained in terms of Abel–Gontcharoff polynomials. The new methodology followed also provides a unified approach to a number of recent works in the literature.

In this paper, we propose a new urn model. A single urn contains b black balls and w white balls. For each observation, we randomly draw m balls and note their colors, say k black balls and m − k white balls. We return the drawn balls to the urn with an additional ck black balls and c(m − k) white balls. We repeat this procedure n times and denote by Xn the fraction of black balls after the nth draw. To investigate the asymptotic properties of Xn, we first perform some computational studies. We then show that {Xn} forms a martingale, which converges almost surely to a random variable X. The distribution of X is then shown to be absolutely continuous.

Competing patterns are compound patterns that compete to be the first to occur pattern-specific numbers of times. They represent a generalisation of the sooner waiting time problem and of start-up demonstration tests with both acceptance and rejection criteria. Through the use of finite Markov chain imbedding, the waiting time distribution of competing patterns in multistate trials that are Markovian of a general order is derived. Also obtained are probabilities that each particular competing pattern will be the first to occur its respective prescribed number of times, both in finite time and in the limit.

We extend the definition of level-crossing ordering of stochastic processes, proposed by Irle and Gani (2001), to the case in which the times to exceed levels are compared using an arbitrary stochastic order, and work, in particular, with integral stochastic orders closed for convolution. Using a sample-path approach, we establish level-crossing ordering results for the case in which the slower of the processes involved in the comparison is skip-free to the right. These results are specially useful in simulating processes that are ordered in level crossing, and extend results of Irle and Gani (2001), Irle (2003), and Ferreira and Pacheco (2005) for skip-free-to-the-right discrete-time Markov chains, semi-Markov processes, and continuous-time Markov chains, respectively.

For uniformly ergodic Markov chains, we obtain new perturbation bounds which relate the sensitivity of the chain under perturbation to its rate of convergence to stationarity. In particular, we derive sensitivity bounds in terms of the ergodicity coefficient of the iterated transition kernel, which improve upon the bounds obtained by other authors. We discuss convergence bounds that hold in the case of finite state space, and consider numerical examples to compare the accuracy of different perturbation bounds.

In this paper, we investigate the geometric growth of homogeneous multitype Markov chains whose states have nonnegative integer coordinates. Such models are considered in a situation similar to the supercritical case for branching processes. Finally, our general theoretical results are applied to a class of controlled multitype branching process in which the control is random.

A simple, widely applicable method is described for determining factorial moments of N̂t, the number of occurrences in (0,t] of some event defined in terms of an underlying Markov renewal process, and asymptotic expressions for these moments as t → ∞. The factorial moment formulae combine to yield an expression for the probability generating function of N̂t, and thereby further properties of such counts. The method is developed by considering counting processes associated with events that are determined by the states at two successive renewals of a Markov renewal process, for which it both simplifies and generalises existing results. More explicit results are given in the case of an underlying continuous-time Markov chain. The method is used to provide novel, probabilistically illuminating solutions to some problems arising in the stochastic modelling of ion channels.

Consider two sequences of bounded random variables, a value and a timing process, that satisfy the large deviation principle (LDP) with rate function J(⋅,⋅) and whose cumulative process satisfies the LDP with rate function I(⋅). Under mixing conditions, an LDP for estimates of I constructed by transforming an estimate of J is proved. For the case of a cumulative renewal process it is demonstrated that this approach is favourable to a more direct method, as it ensures that the laws of the estimates converge weakly to a Dirac measure at I.

Particle filters are Monte Carlo methods that aim to approximate the optimal filter of a partially observed Markov chain. In this paper, we study the case in which the transition kernel of the Markov chain depends on unknown parameters: we construct a particle filter for the simultaneous estimation of the parameter and the partially observed Markov chain (adaptive estimation) and we prove the convergence of this filter to the correct optimal filter, as time and the number of particles go to infinity. The filter presented here generalizes Del Moral's Monte Carlo particle filter.

A cut-off phenomenon is shown to occur in a sample of n independent, identically distributed Ornstein-Uhlenbeck processes and its average. Their distributions stay far from equilibrium before a certain O(log(n)) time, and converge exponentially fast after. Precise estimates show that the total variation distance drops from almost 1 to almost 0 over an interval of time of length O(1) around log(n)/(2α), where α is the viscosity coefficient of the sampled process. The distribution of the hitting time of 0 by the average of the sample is computed. As n tends to infinity, the hitting time becomes concentrated around the cut-off instant, and its tails match the estimates given for the total variation distance.

Given an ℝd-valued supercritical branching Wiener process, let ψ(A, T) be the number of particles in A ⊂ ℝd at time T (T = 0, 1, 2, …). We provide a complete asymptotic expansion of ψ(A, T) as T → ∞, generalizing the work of X. Chen.

We introduce the idea of controlling branching processes by means of another branching process, using the fractional thinning operator of Steutel and van Harn. This idea is then adapted to the model of alternating branching, where two Markov branching processes act alternately at random observation and treatment times. We study the extinction probability and limit theorems for reproduction by n cycles, as n → ∞.

In this paper, we introduce a generalization of the two-color multitype contact process intended to mimic a biological process called allelopathy. To be precise, we have two types of particle. Particles of each type give birth to particles of the same type, and die at rate 1. When a particle of type 1 dies, it gives way to a frozen site that blocks particles of type 2 for an exponentially distributed amount of time. Specifically, we investigate in detail the phase transitions and the duality properties of the interacting particle system.

We derive a large deviation principle for a Brownian immigration branching particle system, where the immigration is governed by a Poisson random measure with a Lebesgue intensity measure.

We define an inverse subordinator as the passage times of a subordinator to increasing levels. It has previously been noted that such processes have many similarities to renewal processes. Here we present an expression for the joint moments of the increments of an inverse subordinator. This is an analogue of a result for renewal processes. The main tool is a theorem on processes which are both renewal processes and Cox processes.

Consider the single-server queue with an infinite buffer and a first-in–first-out discipline, either of type M/M/1 or Geom/Geom/1. Denote by 𝒜 the arrival process and by s the services. Assume the stability condition to be satisfied. Denote by 𝒟 the departure process in equilibrium and by r the time spent by the customers at the very back of the queue. We prove that (𝒟, r) has the same law as (𝒜, s), which is an extension of the classical Burke theorem. In fact, r can be viewed as the sequence of departures from a dual storage model. This duality between the two models also appears when studying the transient behaviour of a tandem by means of the Robinson–Schensted–Knuth algorithm: the first and last rows of the resulting semistandard Young tableau are respectively the last instant of departure from the queue and the total number of departures from the store.

We consider the planar random motion of a particle that moves with constant finite speed c and, at Poisson-distributed times, changes its direction θ with uniform law in [0, 2π). This model represents the natural two-dimensional counterpart of the well-known Goldstein–Kac telegraph process. For the particle's position (X(t), Y(t)), t > 0, we obtain the explicit conditional distribution when the number of changes of direction is fixed. From this, we derive the explicit probability law f(x, y, t) of (X(t), Y(t)) and show that the density p(x, y, t) of its absolutely continuous component is the fundamental solution to the planar wave equation with damping. We also show that, under the usual Kac condition on the velocity c and the intensity λ of the Poisson process, the density p tends to the transition density of planar Brownian motion. Some discussions concerning the probabilistic structure of wave diffusion with damping are presented and some applications of the model are sketched.