The theory of allocation indices for defining the optimal policy in multi-armed bandit problems developed by Gittins is presented in the continuous-time case where the projects (or ‘arms’) are strong Markov processes. Complications peculiar to the continuous-time case are discussed. This motivates investigation of whether approximation of the continuous-time problems by discrete-time versions provides a valid technique with convergent allocation indices and optimal expected rewards. Conditions are presented under which the convergence holds.

The paper considers stability and instability properties of the Markov chain generated by the composition of an i.i.d. sequence of random transformations. The transformations are assumed to be either linear mappings or else mappings which can be well approximated near 0 by linear mappings. The main results concern the risk probabilities that the Markov chain enters or exits certain balls centered at 0. An application is given to the probability of extinction in a model from population dynamics.

Consider a real-valued discrete-time stationary and ergodic stochastic process, called the noise process. For each dimension n, we can choose a stationary point process in ℝn and a translation invariant tessellation of ℝn. Each point is randomly displaced, with a displacement vector being a section of length n of the noise process, independent from point to point. The aim is to find a point process and a tessellation that minimizes the probability of decoding error, defined as the probability that the displaced version of the typical point does not belong to the cell of this point. We consider the Shannon regime, in which the dimension n tends to ∞, while the logarithm of the intensity of the point processes, normalized by dimension, tends to a constant. We first show that this problem exhibits a sharp threshold: if the sum of the asymptotic normalized logarithmic intensity and of the differential entropy rate of the noise process is positive, then the probability of error tends to 1 with n for all point processes and all tessellations. If it is negative then there exist point processes and tessellations for which this probability tends to 0. The error exponent function, which denotes how quickly the probability of error goes to 0 in n, is then derived using large deviations theory. If the entropy spectrum of the noise satisfies a large deviations principle, then, below the threshold, the error probability goes exponentially fast to 0 with an exponent that is given in closed form in terms of the rate function of the noise entropy spectrum. This is obtained for two classes of point processes: the Poisson process and a Matérn hard-core point process. New lower bounds on error exponents are derived from this for Shannon's additive noise channel in the high signal-to-noise ratio limit that hold for all stationary and ergodic noises with the above properties and that match the best known bounds in the white Gaussian noise case.

The polydisperse coagulation equation models irreversible aggregation of particles with varying masses. This paper uses a one-parameter family of discrete-time continuous multitype branching processes to solve the polydisperse coagulation equation when

The critical time tc when diverges corresponds to a critical branching process, while post-critical times t> tc correspond to supercritical branching processes.

A generalized semi-Markov process with reallocation (RGSMP) was introduced to accommodate a large class of stochastic processes which cannot be analyzed by the well-known model of an ordinary generalized semi-Markov process (GSMP). For stationary RGSMP whose initial distribution has a product form, we show that, for a randomly chosen clock of a fixed insensitive type, if the lifetime of this clock is changed to infinity, then the background process is stationary under a certain time change. This implies that the expected time required for the tagged clock to consume a given amount x of resource, called the attained sojourn time, is a linear function of x. Such stationarity and linearity results are known for two special RGSMPs: ordinary GSMP and Kelly's symmetric queue. Our results not only extend them to a general RGSMP but also give more detailed formulas, which allow us to calculate for instance the expected attained sojourn time while the background process is in a given state. Furthermore, we remark that analogous results hold for GSMP with point-process input, in which the lifetimes of clocks of a fixed type form an arbitrary stationary sequence (of not necessarily independent random variables).

Some consequences of restarting stochastic search algorithms are studied. It is shown under reasonable conditions that restarting when certain patterns occur yields probabilities that the goal state has not been found by the nth epoch which converge to zero at least geometrically fast in n. These conditions are shown to hold for restarted simulated annealing employing a local generation matrix, a cooling schedule Tn ∼ c/n and restarting after a fixed number r + 1 of duplications of energy levels of states when r is sufficiently large. For simulated annealing with logarithmic cooling these probabilities cannot decrease to zero this fast. Numerical comparisons between restarted simulated annealing and several modern variations on simulated annealing are also presented and in all cases the former performs better.

Two non-parametric methods for the estimation of the directional measure of stationary line and fibre processes in d-dimensional space are presented. The input data for both methods are intersection counts with finitely many test windows situated in hyperplanes. The first estimator is a measure valued maximum likelihood estimator, if applied to Poisson line processes. The second estimator uses an approximation of the associated zonoid (the Steiner compact) by zonotopes. Consistency of both estimators is proved (without use of the Poisson assumption). The estimation methods are compared empirically by simulation.

An abstract gambler's problem is formulated in a continuous-time setting and analogues are proved for some of the discrete-time results of Dubins and Savage in their book How to Gamble if You Must. Applications are made to problems of controlling a Brownian motion process.

An asset manager invests the savings of some investors in a portfolio of defaultable bonds. The manager pays the investors coupons at a constant rate and receives a management fee proportional to the value of the portfolio. He/she also has the right to walk out of the contract at any time with the net terminal value of the portfolio after payment of the investors' initial funds, and is not responsible for any deficit. To control the principal losses, investors may buy from the manager a limited protection which terminates the agreement as soon as the value of the portfolio drops below a predetermined threshold. We assume that the value of the portfolio is a jump diffusion process and find an optimal termination rule of the manager with and without protection. We also derive the indifference price of a limited protection. We illustrate the solution method on a numerical example. The motivation comes from the collateralized debt obligations.

Large deviation principles and related results are given for a class of Markov chains associated to the ‘leaves' in random recursive trees and preferential attachment random graphs, as well as the ‘cherries’ in Yule trees. In particular, the method of proof, combining analytic and Dupuis–Ellis-type path arguments, allows for an explicit computation of the large deviation pressure.

In this study we assume an inventory system for perishable commodities in which the lifetimes of the items stored are i.i.d. random variable with finite mean.

We utilize the analogy between this inventory system and a queueing system with impatient customers, to study the process of the lost demand, the death of the unused items and the number of items in the system.

In this paper we consider a network of ·/GI/1/N queues with finite-length buffers. A suitable Markov process for the time-evolution of this system is defined. This process is subsequently shown to be ergodic under the conditions of Borovkov (1987).

A new class of reliability point-process models for dependent components is introduced. The dependence is expressed through a regression, following a form suggested by Cox (1972) for survival data analysis involving the current life-length of the components. After formulating the current-life process as a Markov process with stationary transitions and stating some general results on asymptotic behavior, we describe the stationary distributions in some bivariate examples. Finally, we discuss statistical inference for the new models, exhibiting and justifying full- and partial-likelihood methods for their analysis.

In this work we analyze a class of 2 × 2 Pólya-Eggenberger urn models with ball replacement matrix and c = pa with . We determine limiting distributions by obtaining a precise recursive description of the moments of the considered random variables, which allows us to deduce asymptotic expansions of the moments. In particular, we obtain limiting distributions for the pills problem a = c = d = 1, originally proposed by Knuth and McCarthy. Furthermore, we also obtain limiting distributions for the well-known sampling without replacement urn, a = d = 1 and c = 0, and generalizations of it to arbitrary and c = 0. Moreover, we obtain a recursive description of the moment sequence for a generalized problem.

We consider the problem of estimation in a partially observed discrete-time Galton-Watson branching process, focusing on the first two moments of the offspring distribution. Our study is motivated by modelling the counts of new cases at the onset of a stochastic epidemic, allowing for the facts that only a part of the cases is detected, and that the detection mechanism may affect the evolution of the epidemic. In this setting, the offspring mean is closely related to the spreading potential of the disease, while the second moment is connected to the variability of the mean estimators. Inference for branching processes is known for its nonstandard characteristics, as compared with classical inference. When, in addition, the true process cannot be directly observed, the problem of inference suffers significant further perturbations. We propose nonparametric estimators related to those used when the underlying process is fully observed, but suitably modified to take into account the intricate dependence structure induced by the partial observation and the interaction scheme. We show consistency, derive the limiting laws of the estimators, and construct asymptotic confidence intervals, all valid conditionally on the explosion set.

In a branching process with variable lifetime, introduced by Fildes (1972) define Yjk(t) as the number of particles alive in generation k at time t when the initial particle is born in generation j. A limit law similar to that derived in the Bellman-Harris process is proved where it is shown that Yjk(t) suitably normalised converges in mean square to a random variable which is the limit random variable of Znm–n in the Galton-Watson process (m is the mean number of particles born).

We derive results that show the impact of aggregation in a queueing network. Our model consists of a two-stage queueing system where the first (upstream) queue serves many flows, of which a certain subset arrive at the second (downstream) queue. The downstream queue experiences arbitrary interfering traffic. In this setup, we prove that, as the number of flows being aggregated in the upstream queue increases, the overflow probability of the downstream queue converges uniformly in the buffer level to the overflow probability of a single queueing system obtained by simply removing the upstream queue in the original two-stage queueing system. We also provide the speed of convergence and show that it is at least exponentially fast. We then extend our results to non-i.i.d. traffic arrivals.

Batch departures arise in various applications of queues. In particular, such models have been studied recently in connection with production systems. For the most part, however, these models assume Poisson arrivals and exponential service times; little is known about them under more general settings. We consider how their stationary queue length distributions are affected by the distributions of interarrival times, service times and departing batch sizes of customers. Since this is not an easy problem even for single departure models, we first concentrate on single-node queues with a symmetric service discipline, which is known to have nice properties. We start with pre-emptive LIFO, a special case of the symmetric service discipline, and then consider symmetric queues with Poisson arrivals. Stability conditions and stationary queue length distributions are obtained for them, and several stochastic order relations are considered. For the symmetric queues and Poisson arrivals, we also discuss their network. Stability conditions and the stationary joint queue length distribution are obtained for this network.

The third-generation (3G) mobile communication system uses a technique called code division multiple access (CDMA), in which multiple users use the same frequency and time domain. The data signals of the users are distinguished using codes. When there are many users, interference deteriorates the quality of the system. For more efficient use of resources, we wish to allow more users to transmit simultaneously, by using algorithms that utilize the structure of the CDMA system more effectively than the simple matched filter (MF) system used in the proposed 3G systems. In this paper, we investigate an advanced algorithm called hard-decision parallel interference cancellation (HD-PIC), in which estimates of the interfering signals are used to improve the quality of the signal of the desired user. We compare HD-PIC with MF in a simple case, where the only two parameters are the number of users and the length of the coding sequences. We focus on the exponential rate for the probability of a bit-error, explain the relevance of this parameter, and investigate how it scales when the number of users grows large. We also review extensions of our results, proved elsewhere, showing that in HD-PIC, more users can transmit without errors than in the MF system.