Hostname: page-component-586b7cd67f-t7fkt Total loading time: 0 Render date: 2024-11-28T02:28:18.233Z Has data issue: false hasContentIssue false
  • English
  • Français

The fundamental matrix of singularly perturbed Markov chains

Part of: Markov processes General theory of linear operators

Published online by Cambridge University Press:  01 July 2016

Konstantin E. Avrachenkov  and
Jean B. Lasserre
Konstantin E. Avrachenkov*
Affiliation:
University of South Australia
Jean B. Lasserre*
Affiliation:
LAAS-CNRS
*
∗ Postal address: School of Mathematics, University of South Australia, The Levels, SA 5095, Australia.
∗∗ Postal address: 7 Avenue du Colonel Roche, 31 077 Toulouse, Cédex 4, France. Email address: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

We consider a singularly perturbed (finite state) Markov chain and provide a complete characterization of the fundamental matrix. In particular, we obtain a formula for the regular part simpler than a previous formula obtained by Schweitzer, and the singular part is obtained via a reduction process similar to Delebecque's reduction for the stationary distribution. In contrast to previous approaches, one works with aggregate Markov chains of much smaller dimension than the original chain, an essential feature for practical computation. An application to mean first-passage times is also presented.

Keywords

Markov chainsfundamental matrixsingular perturbation

MSC classification

Primary: 60J10: Markov chains (discrete-time Markov processes on discrete state spaces) 47A55: Perturbation theory
Secondary: 60J35: Transition functions, generators and resolvents
Type
General Applied Probability
Information
Advances in Applied Probability , Volume 31 , Issue 3 , September 1999 , pp. 679 - 697
Copyright
Copyright © Applied Probability Trust 1999 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Part of this work was done while the second author was visiting the University of South Australia and supported by the Australian Research Council under Grant A49532206.

References

Abbad, M., Filar, J. A. and Bielecki, T. R. (1992). Algorithms for singularly perturbed limiting average Markov control problems. IEEE Trans. Autom. Control 37, 1421–1425.Google Scholar
Baumgartel, H. (1985). Analytic Perturbation Theory for Matrices and Operators. Birkhäuser, Boston, MA.Google Scholar
Bielecki, T. R. and Filar, J. A. (1991). Singularly perturbed Markov control problem: limiting average cost. Annals Operat. Res. 28, 153–168.Google Scholar
Chen, M. and Filar, J. A. (1992). Hamiltonian cycles, quadratic programming and Markov chains. In Recent Advances in Global Optimization. Princeton University Press, NJ, pp. 32–49.Google Scholar
Coderch, M., Willsky, A. S., Sastry, S. S. and Castanon, D. A. (1983). Hierarchical aggregation of singularly perturbed finite state Markov processes. Stochastics 8, 259–289.CrossRefGoogle Scholar
Courtois, P. J. and Louchard, G., (1976). Approximation of eigencharacteristics in nearly completely decomposable systems. Stoch. Proc. Appl. 4, 283–296.Google Scholar
Delebecque, F. and Quadrat, J. P. (1981). Optimal control of Markov chains admitting strong and weak interactions. Automatica 17, 281–296.CrossRefGoogle Scholar
Delebecque, F. (1983). A reduction process for perturbed Markov chains. SIAM J. Appl. Math. 48, 325–350.CrossRefGoogle Scholar
Filar, J. A. and Liu, K. (1996). Hamiltonian cycle problem and singularly perturbed Markov decision process. Preprint.Google Scholar
Gaitsgori, V. G. and Pervozvanskii, A. A. (1975). Aggregation of states in a Markov chain with weak interactions. Cybernetics 11, 441–450. (Translation of Russian original (1975) Kibernetika 11, 91–98.)Google Scholar
Hassin, R. and Haviv, M. (1992). Mean passage times and nearly uncoupled Markov chains. SIAM J. Disc. Math. 5, 386–397.Google Scholar
Haviv, M. and Ritov, Y. (1993). On series expansions and stochastic matrices. SIAM J. Matrix Anal. Appl. 14, 670–676.Google Scholar
Howlett, P. G. (1982). Input retrieval in finite dimensional linear systems. J. Austral. Math. Soc. 23, 357–382.CrossRefGoogle Scholar
Iosifescu, M. (1980). Finite Markov Processes and Their Applications. Wiley, Bucuresti.Google Scholar
Kato, T. (1980). Perturbation Theory for Linear Operators. Springer, Berlin.Google Scholar
Kemeny, J. G. and Snell, J. L. (1976). Finite Markov Chains. Springer, New York.Google Scholar
Korolyuk, V. S. and Turbin, A. F. (1978). Mathematical Foundations of the State Lumping of Large Systems. Naukova Dumka, Kiev (in Russian).Google Scholar
Langenhop, C. E. (1971). The Laurent expansion for a nearly singular matrix. Linear Algebra Appl. 4, 329–340.Google Scholar
Lasserre, J. B. (1994). A formula for singular perturbation of Markov chains. J. Appl. Prob. 31, 829–833.Google Scholar
Latouche, G. and Louchard, G. (1978). Return times in nearly completely decomposable stochastic processes. J. Appl. Prob. 15, 251–267.Google Scholar
Louchard, G. and Latouche, G. (1990). Geometric bounds on iterative approximations for nearly completely decomposable Markov chains. J. Appl. Prob. 27, 521–529.Google Scholar
Pervozvanskii, A. A. and Gaitsgori, V. G. (1988). Theory of Suboptimal Decisions. Kluwer, Dordrecht, Netherlands.CrossRefGoogle Scholar
Pervozvanskii, A. A. and Smirnov, I. N. (1974). Stationary-state evaluation for a complex system with slowly varying couplings. Cybernetics 10, 603–611. (Translation of Russian original (1974) Kibernetika 10, 45–51.)Google Scholar
Phillips, R. G. and Kokotovic, P. V. (1981). A singular perturbation approach to modeling and control of Markov chains. IEEE Trans. Automat. Control 26, 1087–1094.Google Scholar
Rohlicek, J. R. and Willsky, A. S. (1988). The reduction of Markov generators: an algorithm exposing the role of transient states. J. Assoc. Comput. Mach. 35, 675–696.CrossRefGoogle Scholar
Sain, M. K. and Massey, J. L. (1969). Invertibility of linear-time invariant dynamical systems. IEEE Trans. Automat. Control 14, 141–149.CrossRefGoogle Scholar
Schweitzer, P. J. (1968). Perturbation theory and finite Markov chains. J. Appl. Prob. 5, 401–413.Google Scholar
Schweitzer, P. J. (1981). Perturbation series expansions of nearly completely-decomposable Markov chains. Working paper Series No. 8122, Graduate School of Management, University of Rochester.Google Scholar
Schweitzer, P. J. (1986). Perturbation series expansions for nearly completely-decomposable Markov chains. In Teletrafic Analysis and Computer Performance Evaluation, eds. Boxma, O. J., Cohen, J. W. and Tijms, H. C., pp. 319–328. Elsevier (North-Holland), Amsterdam.Google Scholar
Schweitzer, P. J. and Stewart, G. (1993). The Laurent expansion of pencils that are singular at the origin. Linear Algebra Appl. 183, 237–254.CrossRefGoogle Scholar
Simon, H. A. and Ando, A. (1961). Aggregation of variables in dynamic systems. Econometrica 29, 111–138.CrossRefGoogle Scholar