Abstract
This note studies the variance of total cumulative rewards for Markov reward chains in both discrete and continuous time. It is shown that parallel results can be obtained for both cases.
First, explicit formulae are presented for the variance within finite time. Next, the infinite time horizon is considered. Most notably, it is concluded that the variance has a linear growth rate. Explicit expressions are provided, related to the standard average reward case, to compute this growth rate.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Benito, F. (1982): Calculating the variance in Markov processes with random reward. Trabajos de Estadistica y de Investigacion Operativa, 33, 73–85
Mandl, P. (1971): On the variance in controlled Markov chains. Kybernetika, 7, 1–12
Puterman, M. L. (1994): Markov Decision Processes — Discrete Stochastic Dynamic Programming. Wiley, New York
Ross, S.M. (1970): Applied Probability Models with Optimization Applications. Holden-Day, San Francisco, CA
Sladký, K., Sitař, M. (2004): Optimal solutions for undiscounted variance penalized Markov decision chains. In: Dynamic Stochastic Optimization (Marti, K., Ermoliev, Y., Pflug, G., Eds.), LNEMS, Vol. 532, Springer, Berlin, pp. 43–66
Sobel, M. J. (1982): The variance of discounted Markov decision processes. J. Appl. Probab., 19, 794–802
van Dijk, N. M., Sladky, K. (2004): On total reward variance in continuous-time Markov reward chains. Manuscript
White, D. J. (1988): Mean, variance and probability criteria in finite Markov decision processes: A review. J. Optim. Theory Appl., 56, 1–29
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sladký, K., van Dijk, N.M. (2005). Total Reward Variance in Discrete and Continuous Time Markov Chains. In: Fleuren, H., den Hertog, D., Kort, P. (eds) Operations Research Proceedings 2004. Operations Research Proceedings, vol 2004. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-27679-3_40
Download citation
DOI: https://doi.org/10.1007/3-540-27679-3_40
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-24274-1
Online ISBN: 978-3-540-27679-1
eBook Packages: Business and EconomicsBusiness and Management (R0)