Motivated by a class of Partially Observable Markov Decision Processes with application in surveillance systems in which a set of imperfectly observed state processes is to be inferred from a subset of available observations through a Bayesian approach, we formulate and analyze a special family of multi-armed restless bandit problems. We consider the problem of finding an optimal policy for observing the processes that maximizes the total expected net rewards over an infinite time horizon subject to the resource availability. From the Lagrangian relaxation of the original problem, an index policy can be derived, as long as the existence of the Whittle index is ensured. We demonstrate that such a class of reinitializing bandits in which the projects' state deteriorates while active and resets to its initial state when passive until its completion possesses the structural property of indexability and we further show how to compute the index in closed form. In general, the Whittle index rule for restless bandit problems does not achieve optimality. However, we show that the proposed Whittle index rule is optimal for the problem under study in the case of stochastically heterogenous arms under the expected total criterion, and it is further recovered by a simple tractable rule referred to as the 1-limited Round Robin rule. Moreover, we illustrate the significant suboptimality of other widely used heuristic: the Myopic index rule, by computing in closed form its suboptimality gap. We present numerical studies which illustrate for the more general instances the performance advantages of the Whittle index rule over other simple heuristics.

We consider a monotone binary system with ternary components. “Ternary” means that each component can be in one of three states: up, middle (mid) and down. Handling such systems is a hard task, even if a part of the components have no mid state. Nevertheless, the permutation Monte Carlo methods, that proved very useful for dealing with binary components, can be efficiently used also for ternary monotone systems. It turns out that for “ternary” system there also exists a combinatorial invariant by means of which it becomes possible to count the number C(r;x) of system failure sets which have a given number r and x of components in up and down states, respectively. This invariant is called ternary D-spectrum and it is an analogue of the D-spectrum (or signature) of a system with binary components. Its value is the knowledge of system failure or path set properties which do not depend on stochastic mechanism governing component failures. In case of independent and identical components, knowing D-spectrum makes it easy to calculate system UP or DOWN probability for a variety of UP/DOWN definitions suitable for systems of many types, like communication networks, flow and supply networks, etc.

Consider a two-station, heterogeneous parallel queueing system in which each station operates as an independent M/M/1 queue with its own infinite-capacity buffer. The input to the system is a Poisson process that splits among the two stations according to a Bernoulli splitting mechanism. However, upon arrival, a strategic customer initially joins one of the queues selectively and decides at subsequent arrival and departure epochs whether to jockey (or switch queues) with the aim of reducing her own sojourn time. There is a holding cost per unit time, and jockeying incurs a fixed non-negative cost while placing the customer at the end of the other queue. We examine individually optimal joining and jockeying policies that minimize the strategic customer's total expected discounted (or undiscounted) costs over finite and infinite time horizons. The main results reveal that, if the strategic customer is in station 1 with ℓ customers in front of her, and q1 and q2 customers in stations 1 and 2, respectively (excluding herself), then the incentive to jockey increases as either ℓ increases or q2 decreases. Numerical examples reveal that it may not be optimal to join, and/or jockey to, the station with the shortest queue or the fastest server.

We study the optimal buffer capacity K for the M/M/1/K queue under some standard cost and reward structures by comparing various Markov reward processes. Using explicit expressions for the deviation matrix of the underlying Markov chains, we find the bias optimal value for K in the case of a tie between two consecutive optimal gain policies. We show that the bias optimal value depends both on whether the reward is granted upon arrival or departure of the customers, and on the initial queue size. Moreover, we demonstrate that in some specific cases the optimal policy is threshold-based with respect to the initial queue size.

In this paper, we study birth/immigration-death processes under mild (binomial) catastrophes. We obtain explicit expressions for both the time-dependent (transient) and the limiting (equilibrium) factorial moments, which are then used to construct the transient and equilibrium distribution of the population size. We demonstrate that our approach is also applicable to multidimensional systems such as stochastic processes operating under a random environment and other variations of the model at hand. We also obtain various stochastic order results for the number of individuals with respect to the system parameters, as well as the relaxation time.

We revisit the random m-ary search tree and study a finer profile of its node outdegrees with the purpose of exploring possibilities of data structure compression. The analysis is done via Pólya urns. The analysis shows that the number of nodes of each individual node outdegree has a phase transition: Up to m = 26, the number of nodes of outdegree k, for k = 0, 1, …, m, is asymptotically normal; that behavior changes at m = 27. Based on the analysis, we propose a compact m-ary tree that offers significant space saving.

De Bruijn's identity shows a link between two fundamental concepts in information theory: entropy and Fisher information. In the literature, De Bruijn's identity has been stated under the assumption of independence between input signal and an additive noise. However, in the real world, the noise could be highly dependent on the main signal. The main aim of this paper is, firstly, to extend De bruijn's identity for signal-dependent noise channels and, secondly, to study how Stein and heat identities are related to De bruijn's identity. Thus, new versions of De Bruijn's identity are introduced in the case when input signal and additive noise are dependent and are jointly distributed according to Archimedean and Gaussian copulas. It is shown that in this generalized model, the derivatives of the differential entropy can be expressed in terms of a function of Fisher information. Our finding enfolds the conventional De Bruijn's identity as some remarks. Then, the equivalence among the new De Bruijn-type identity, Stein's identity and heat equation identity is established. The paper concludes with an application of the developed results in information theory.