Hostname: page-component-78c5997874-lj6df Total loading time: 0 Render date: 2024-11-03T08:17:11.507Z Has data issue: false hasContentIssue false
  • English
  • Français

IMPRECISE MARKOV CHAINS AND THEIR LIMIT BEHAVIOR

Published online by Cambridge University Press:  04 August 2009

Gert de Cooman ,
Filip Hermans  and
Erik Quaeghebeur
Gert de Cooman
Affiliation:
SYSTeMS Research Group, Ghent University, Technologiepark–Zwijnaarde 914, 9052 Zwijnaarde, Belgium E-mail: [email protected]; [email protected]; [email protected]
Filip Hermans
Affiliation:
SYSTeMS Research Group, Ghent University, Technologiepark–Zwijnaarde 914, 9052 Zwijnaarde, Belgium E-mail: [email protected]; [email protected]; [email protected]
Erik Quaeghebeur
Affiliation:
SYSTeMS Research Group, Ghent University, Technologiepark–Zwijnaarde 914, 9052 Zwijnaarde, Belgium E-mail: [email protected]; [email protected]; [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

When the initial and transition probabilities of a finite Markov chain in discrete time are not well known, we should perform a sensitivity analysis. This can be done by considering as basic uncertainty models the so-called credal sets that these probabilities are known or believed to belong to and by allowing the probabilities to vary over such sets. This leads to the definition of an imprecise Markov chain. We show that the time evolution of such a system can be studied very efficiently using so-called lower and upper expectations, which are equivalent mathematical representations of credal sets. We also study how the inferred credal set about the state at time n evolves as n→∞: under quite unrestrictive conditions, it converges to a uniquely invariant credal set, regardless of the credal set given for the initial state. This leads to a non-trivial generalization of the classical Perron–Frobenius theorem to imprecise Markov chains.

Type
Research Article
Information
Probability in the Engineering and Informational Sciences , Volume 23 , Issue 4 , October 2009 , pp. 597 - 635
Copyright
Copyright © Cambridge University Press 2009

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

1.Avis, D., Bremner, D. & Seidel, R. (1997). How good are convex hull algorithms? Computational Geometry 7(5–6): 265–301.CrossRefGoogle Scholar
2.Campos, M.A., Dimuro, G.P., da Rocha Costa, A.C. & Kreinovich, V. (2003). Computing 2-step predictions for interval-valued finite stationary Markov chains. Technical report UTEP-CS-03-20a, University of Texas at El Paso.Google Scholar
3.de Campos, L.M., Huete, J.F. & Moral, S. (1994). Probability intervals: A tool for uncertain reasoning. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 2: 167–196.CrossRefGoogle Scholar
4.Cozman, F.G. (2000). Credal networks. Artificial Intelligence 120: 199–233.Google Scholar
5.Cozman, F.G. (2005). Graphical models for imprecise probabilities. International Journal of Approximate Reasoning 39(2–3): 167–184.CrossRefGoogle Scholar
6.De Cooman, G. & Hermans, F. (2008). Imprecise probability trees: Bridging two theories of imprecise probability. Artificial Intelligence 172(11): 1400–1427.Google Scholar
7.De Cooman, G. & Miranda, E. (2007). Symmetry of models versus models of symmetry. In Harper, W.L. & Wheeler, G.R., (eds.) Probability and inference: Essays in honor of Henry E. Kyburg, Jr. London: King's College Publications, pp. 67–149.Google Scholar
8.De Cooman, G., Troffaes, M.C.M. & Miranda, E. (2008). n-Monotone exact functionals. Journal of Mathematical Analysis and Applications 347: 143–156.Google Scholar
9.de Finetti, B. (1970). Teoria delle Probabilità. Turin: Einaudi.Google Scholar
10.de Finetti, B. (1974–1975). Theory of probability: A critical introductory treatment. Chichester: Wiley. English translation of [9], two volumes.Google Scholar
11.Dhaenens, S. (2007). Onderzoek van imprecieze Markov-modellen. Master's thesis, Ghent University.Google Scholar
12.Harmanec, D. (2002). Generalizing Markov decision processes to imprecise probabilities. Journal of Statistical Planning and Inference 105: 199–213.Google Scholar
13.Hartfiel, D.J. (1991). Sequential limits in Markov set-chains. Journal of Applied Probability 28(4): 910–913.Google Scholar
14.Hartfiel, D.J. (1998). Markov set-chains. Lecture Notes in Mathematics No, 1695. Berlin: Springer.CrossRefGoogle Scholar
15.Hartfiel, D.J. & Seneta, E. (1994). On the theory of Markov set-chains. Advances in Applied Probability 26: 947–964.CrossRefGoogle Scholar
16.Huygens, C. (1656–1657). Van Rekeningh in Spelen van Geluck. Reprinted in Volume XIV of [17].Google Scholar
17.Huygens, C. (1888–1950). Oeuvres complètes de Christiaan Huygens. Den Haag: Martinus Nijhoff. Twenty-two volumes. Available in digitized form from the Bibliothèque nationale de France (http://gallica.bnf.fr).Google Scholar
18.Itoh, H. & Nakamura, K. (2007). Partially observable Markov decision processes with imprecise parameters. Artificial Intelligence 171: 453–490.CrossRefGoogle Scholar
19.Kemeny, J.G. & Snell, J.L. (1976). Finite markov chains. Undergraduate Text in Mathematics. New York: Springer-Verlag.Google Scholar
20.Koutras, M.V. (1996). On a Markov chain approach for the study of reliability structures. Journal of Applied Probability 33(2): 357–367.CrossRefGoogle Scholar
21.Kozine, I.O. & Utkin, L.V. (2002). Interval-valued finite Markov chains. Reliable Computing 8(2): 97–113.Google Scholar
22.Luenberger, D.G. (1979). Introduction to dynamic systems. Theory, models & applications. New York: Wiley.Google Scholar
23.Miranda, E. & de Cooman, G. (2007). Marginal extension in the theory of coherent lower previsions. International Journal of Approximate Reasoning 46(1): 188–225.Google Scholar
24.Moral, S. (2005). Epistemic irrelevance on sets of desirable gambles. Annals of Mathematics and Artificial Intelligence 45: 197–214.CrossRefGoogle Scholar
25.Nilim, A. & El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operations Research 53: 780–798.Google Scholar
26.Nussbaum, R.D., Scheutzow, M. & Verduyn Lunel, S.M. (1998). Periodic points of nonexpansive maps and nonlinear generalizations of the Perron–Frobenius theory. Selecta Mathematica 4: 141–181.CrossRefGoogle Scholar
27.Satia, J.K. & Lave, R.E. (1973). Markovian decision processes with uncertain transition probabilities. Operations Research 21: 728–740.CrossRefGoogle Scholar
28.Shafer, G. (1976). A mathematical theory of evidence. Princeton, NJ: Princeton University Press.Google Scholar
29.Shafer, G. (1996). The art of causal conjecture. Cambridge, MA: The MIT Press.Google Scholar
30.Sine, R. (1990). A nonlinear Perron–Frobenius theorem. Proceedings of the American Mathematical Society 109(2): 331–336.Google Scholar
31.Škulj, D. (2006). Finite discrete time Markov chains with interval probabilities. In Lawry, J., Miranda, E., Bugarin, A., Li, S., Gil, M.A., Grzegorzewski, P. & Hryniewicz, O. (eds.) Soft methods for integrated uncertainty modelling. Berlin: Springer, pp. 299–306.Google Scholar
32.Škulj, D. (2007). Regular finite Markov chains with interval probabilities. In de Cooman, G., Vejnarová, J. & Zaffalon, M. (eds.) ISIPTA ’07 — Proceedings of the fifth international symposium on imprecise probability: Theories and applications. Prague: SIPTA, pp. 405–413.Google Scholar
33.Walley, P. (1991). Statistical reasoning with imprecise probabilities. London: Chapman & Hall.Google Scholar
34.Walley, P. (1996). Measures of uncertainty in expert systems. Artificial Intelligence 83(1): 1–58.Google Scholar
35.Weichselberger, K. (2001). Elementare Grundbegriffe einer allgemeineren Wahrscheinlichkeitsrechnung. I. Intervallwahrscheinlichkeit als umfassendes Konzept. Heidelberg: Physica-Verlag.Google Scholar
36.White, C.C. & Eldeib, H.K. (1994). Markov decision-processes with imprecise transition-probabilities. Operations Research 42: 739–749.Google Scholar
37.Whittle, P. (2000). Probability via expectation. 4th ed.New York: Springer.Google Scholar