Article

Free access

A survey of ranking, selection, and multiple comparison procedures for discrete-event simulation

Authors:

James R. Swisher,

Sheldon H. JacobsonAuthors Info & Claims

WSC '99: Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future - Volume 1

Pages 492 - 501

https://doi.org/10.1145/324138.324307

Published: 01 December 1999 Publication History

PDF eReader

References

[1]

Bechhofer, R.E. 1954. A single-sample multiple decision procedure for ranking means of normal populations with known variances. Annals of Mathematical Statistics 25:16-39.

Crossref

Google Scholar

[2]

Bechhofer, R.E., J. Kiefer, and M. Sobel. 1968. Sequential Identification and Ranking Procedures. The University of Chicago Press, Chicago, Illinois.

Google Scholar

[3]

Bechhofer, R.E., T.J. Santner, and D.M. Goldsman. 1995. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. John Wiley & Sons, Inc., New York.

Google Scholar

[4]

Bofinger, E. and G.J. Lewis. 1992. Two-stage procedures for multiple comparisons with a control. American Journal of Mathematical and Management Sciences 12(4): 253-275.

Crossref

Google Scholar

[5]

Bratley, P., B.L. Fox, and L.E. Schrage. 1987. A Guide to Simulation, Second Edition. Springer-Verlag, New York.

Digital Library

Google Scholar

[6]

Chen, C.H. 1995. An effective approach to smartly allocate computing budget for discrete-event simulation. Proceedings of the 34th IEEE Conference on Decision and Control, 2598-2605.

Google Scholar

[7]

Chen, C.H., H.C. Chen, and L. Dai. 1996. A gradient approach for smartly allocating computing budget for discrete event simulation. Proceedings of the 1996 Winter Simulation Conference, 398-405.

Digital Library

Google Scholar

[8]

Chen, C.H., Y. Yuan, H.C. Chen, E. Y~cesan, and L. Dai. 1998. Computing budget allocation for simulation experiments with different system structures. Proceedings of the 1998 Winter Simulation Conference, 735-741.

Digital Library

Google Scholar

[9]

Chen, H.C., C.H. Chen, L. Dai, and E. Y~cesan. 1997. New development of optimal computing budget allocation for discrete event simulation. Proceedings of the 1997 Winter Simulation Conference, 334-341.

Digital Library

Google Scholar

[10]

Chen, P. 1988. On selecting the best of k systems: An expository survey of subset-selection multinomial procedures. Proceedings of the 1988 Winter Simulation Conference, 440-444.

Digital Library

Google Scholar

[11]

Chick, S.E. 1997. Selecting the best system: A decision-theoretic approach. Proceedings of the 1997 Winter Simulation Conference, 326-333.

Digital Library

Google Scholar

[12]

Chick, S.E. and K. Inoue. 1998. Sequential allocations that reduce risk for multiple comparisons. Proceedings of the 1987 Winter Simulation Conference, 669-676.

Digital Library

Google Scholar

[13]

Clark, G.M. and W.N. Yang. 1986. A Bonferroni selection procedure when using common random numbers with unknown variances. Proceedings of the 1986 Winter Simulation Conference, 313-315.

Digital Library

Google Scholar

[14]

Damerdji, H. and M.K. Nakayama. 1996. Two-stage procedures for multiple comparisons with a control in steady-state simulations. Proceedings of the 1996 Winter Simulation Conference, 372-375.

Digital Library

Google Scholar

[15]

Damerdji, H., P.W. Glynn, M.K. Nakayama, and J.R. Wilson. 1996. Selecting the best system in transient simulations with known variances. Proceedings of the 1996 Winter Simulation Conference, 281-286.

Digital Library

Google Scholar

[16]

Dudewicz, E.J. 1976. Statistics in simulation: How to design for selecting the best alternative. Proceedings of the 1976 Winter Simulation Conference, 67-71.

Digital Library

Google Scholar

[17]

Dudewicz, E.J. and S.R. Dalal. 1975. Allocation of observations in ranking and selection with unequal variances. The Indian Journal of Statistics 37B(1):28- 78.

Google Scholar

[18]

Dudewicz, E.J. and V.S. Taneja. 1978. Multivariate ranking and selection without reduction to a univariate problem. Proceedings of the 1978 Winter Simulation Conference, 207-210.

Digital Library

Google Scholar

[19]

Dunnett, C.W. 1955. A multiple comparisons procedure for comparing several treatments with a control. Journal of the American Statistical Association 78:965-971.

Google Scholar

[20]

Dunnett, C.W. 1989. Multivariate normal probability integrals with product correlation structure. Applied Statistics 38:564-579. Correction 42:709.

Crossref

Google Scholar

[21]

Fu, M. 1994a. Optimization via simulation: A review. Annals of Operations Research 53:199-247.

Crossref

Google Scholar

[22]

Fu, M. 1994b. A tutorial review of techniques for simulation optimization. Proceedings of the 1994 Winter Simulation Conference, 149-156.

Digital Library

Google Scholar

[23]

Goldsman, D. 1983. Ranking and selection in simulation. Proceedings of the 1983 Winter Simulation Conference, 387-393.

Digital Library

Google Scholar

[24]

Goldsman, D. 1984a. A multinomial ranking and selection procedure: Simulation and applications. Proceedings of the 1984 Winter Simulation Conference, 259-263.

Digital Library

Google Scholar

[25]

Goldsman, D. 1984b. On selecting the best of k systems: An expository survey of indifference-zone multinomial procedures. Proceedings of the 1984 Winter Simulation Conference, 107-112.

Digital Library

Google Scholar

[26]

Goldsman, D. 1985. Ranking and selection procedures using standardized time series. Proceedings of the 1985 Winter Simulation Conference, 120-123.

Digital Library

Google Scholar

[27]

Goldsman, D. 1986. Tutorial on indifference-zone normal means ranking and selection procedures. Proceedings of the 1986 Winter Simulation Conference, 370-375.

Digital Library

Google Scholar

[28]

Goldsman, D. 1987. Ranking and selection tutorial: 2- factor normal means procedures. Proceedings of the 1987 Winter Simulation Conference, 52-57.

Digital Library

Google Scholar

[29]

Goldsman, D. and B.L. Nelson. 1990. Batch-size effects on simulation optimization using multiple comparisons. Proceedings of the 1990 Winter Simulation Conference, 288-293.

Digital Library

Google Scholar

[30]

Goldsman, D. and B.L. Nelson. 1994. Ranking, selection and multiple comparisons in computer simulation. Proceedings of the 1994 Winter Simulation Conference, 192-199.

Digital Library

Google Scholar

[31]

Goldsman, D. and B.L. Nelson. 1998. Statistical screening, selection, and multiple comparison procedures in computer simulation. Proceedings of the 1998 Winter Simulation Conference, 159-166.

Digital Library

Google Scholar

[32]

Goldsman, D., B.L. Nelson, and B. Schmesier. 1991. Methods for selecting the best system. Proceedings of the 1991 Winter Simulation Conference, 177-186.

Digital Library

Google Scholar

[33]

Gray, D. and D. Goldsman. 1988. Indifference-zone selection procedures for choosing the best airspace configuration. Proceedings of the 1988 Winter Simulation Conference, 445-450.

Digital Library

Google Scholar

[34]

Gupta, S.S. 1956. On a decision rule for a problem in ranking means. Mimeograph Series No. 150, Institute of Statistics. University of North Carolina, Chapel Hill.

Google Scholar

[35]

Gupta, S.S. and J.C. Hsu. 1977. Subset selection procedures with special reference to the analysis of two-way layout: Application to motor-vehicle fatality data. Proceedings of the 1977 Winter Simulation Conference, 81-85.

Digital Library

Google Scholar

[36]

Gupta, S.S. and J.C. Hsu. 1984. A computer package for ranking, selection, and multiple comparisons with the best. Proceedings of the 1984 Winter Simulation Conference, 251-257.

Digital Library

Google Scholar

[37]

Gupta, S.S. and T.J. Santner. 1973. On selection and ranking procedures - a restricted subset selection rule. Proceedings of the 39 th Session of the International Statistical Institute, Volume 1.

Google Scholar

[38]

Ho, Y.C., R. Sreenivas, and P. Vakili. 1992. Ordinal optimization of discrete event dynamic systems. Discrete Event Dynamical Systems, 2(2), 61-88.

Crossref

Google Scholar

[39]

Hochberg, Y. and A.C. Tamhane. 1987. Multiple Comparison Procedures. John Wiley and Sons, New York.

Digital Library

Google Scholar

[40]

Hsu, J.C. 1984. Constrained simultaneous confidence intervals for multiple comparisons with the best. Annals of Statistics 12:1136-1144.

Crossref

Google Scholar

[41]

Hsu, J.C. 1996. Multiple Comparisons: Theory and Methods. Chapman & Hall, London.

Google Scholar

[42]

Hsu, J.C. and B.L. Nelson. 1988. Optimization over a finite number of system designs with one-stage sampling and multiple comparisons with the best. Proceedings of the 1988 Winter Simulation Conference, 451-457.

Digital Library

Google Scholar

[43]

Inoue, K. and S.E. Chick. 1998. Comparison of bayesian and frequentist assessments of uncertainty for selecting the best system. Proceedings of the 1998 Winter Simulation Conference, 727-734.

Digital Library

Google Scholar

[44]

Jacobson, S.H. and L.W. Schruben. 1989. A review of techniques for simulation optimization. Operations Research Letters 8:1-9.

Digital Library

Google Scholar

[45]

Jacobson, S.H., L.W. Schruben, J.R. Swisher, and P. Hyden. 1999. Discrete-event simulation optimization: A survey of recent advances. Submitted for publication.

Google Scholar

[46]

Koenig, L.W. and A.M. Law. 1985. A procedure for selecting a subset of size m containing the l best of k independent normal populations, with applications to simulation. Communications in Statistics B14(3):719- 734.

Crossref

Google Scholar

[47]

Kolojna, B., D.R. Kalasky, and J.M. Mutmansky. 1993. Optimization of dispatching criteria for open-pit truck haulage system design using multiple comparisons with the best and common random numbers. Proceedings of the 1993 Winter Simulation Conference, 393-401.

Digital Library

Google Scholar

[48]

Law, A.M. and W.D. Kelton. 1991. Simulation Modeling and Analysis, Second Edition. McGraw-Hill, Inc., New York.

Digital Library

Google Scholar

[49]

Matejcik, F.J. and B.L. Nelson. 1993. Simultaneous ranking, selection and multiple comparisons for simulation. Proceedings of the 1993 Winter Simulation Conference, 386-392.

Digital Library

Google Scholar

[50]

Matejcik, F.J. and B.L. Nelson. 1995. Two-stage multiple comparisons with the best for computer simulation. Operations Research 43(4):633-640.

Digital Library

Google Scholar

[51]

Morrice, D.J., J. Butler, and P.W. Mullarkey. 1998. An approach to ranking and selection for multiple performance measures. Proceedings of the 1998 Winter Simulation Conference, 719-725.

Digital Library

Google Scholar

[52]

Nakayama, M.K. 1995. Selecting the best system in steady-state simulations using batch means. Proceedings of the 1995 Winter Simulation Conference, 362-366.

Digital Library

Google Scholar

[53]

Nakayama, M.K. 1996. Multiple comparisons with the best in steady-state simulations, Proceedings of the Second International Workshop on Mathematical Methods in Stochastic Simulation and Experimental Design, 230-235.

Google Scholar

[54]

Nakayama, M.K. 1997a. Multiple-comparison procedures for steady-state simulations, Annals of Statistics 25:2433-2450.

Crossref

Google Scholar

[55]

Nakayama, M.K. 1997b. Using common random numbers in two-stage procedures for multiple comparisons with the best for steady-state simulations. Proceedings of the 11th European Simulation Multiconference, 155- 159.

Google Scholar

[56]

Nakayama, M.K. 1999. Multiple comparisons with the best using common random numbers in steady-state simulations. Journal of Statistical Planning and Inference, to appear.

Google Scholar

[57]

Nelson, B.L. 1993. Robust multiple comparison procedures under common random numbers. ACM Transactions on Modeling and Computer Simulation 3(3):225-243.

Digital Library

Google Scholar

[58]

Nelson, B.L. and J.C. Hsu. 1993. Control-variate models of common random numbers for multiple comparisons with the best. Management Science 39(8): 989-1001.

Digital Library

Google Scholar

[59]

Nelson, B.L. and F.J. Matejcik. 1995. Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Management Science 41(12):1935-1945.

Digital Library

Google Scholar

[60]

Nozari, A., S. Arnold, and C. Pegden. 1987. Statistical analysis for use with Schruben and Margolin correlation induction strategy. Operations Research 35:127-139.

Digital Library

Google Scholar

[61]

Paulson, E. 1964. A sequential procedure for selecting the population with the largest mean from k normal populations. Annals of Mathematical Statistics 35:174-180.

Crossref

Google Scholar

[62]

Rinott, Y. 1978. On two-stage selection procedures and related probability-inequalities. Communications in Statistics A7(8):799-811.

Crossref

Google Scholar

[63]

Sanchez, S.M. 1997. It is a far, far better mean I find ~ Proceedings of the 1997 Winter Simulation Conference, 31-38.

Digital Library

Google Scholar

[64]

Santner, T.J. 1975. A restricted subset selection approach to ranking and selection problems. Annals of Statistics 3:334-349.

Crossref

Google Scholar

[65]

Schruben, L.W. and B. Margolin. 1978. Pseudorandom number assignment in statistically designed simulation and distribution sampling experiments. Journal of the American Statistical Association 73:504-525.

Crossref

Google Scholar

[66]

Sullivan, D.W. and J.R. Wilson. 1989. Restricted subset selection procedures for simulation. Operations Research 37:52-71.

Digital Library

Google Scholar

[67]

Swisher, J.R. 1999. Evaluation of the design of a family practice healthcare clinic using discrete-event simulation. Master's Thesis. Virginia Tech, Blacksburg, Virginia.

Google Scholar

[68]

Tew, J.D. and J.R. Wilson. 1994. Estimating simulation metamodels using combined correlation-based variance reduction techniques. IIE Transactions 26:2- 16.

Crossref

Google Scholar

[69]

Tukey, J.W. 1953. The problem of multiple comparisons. Unpublished manuscript.

Google Scholar

[70]

Wen, M.J. and H.J. Chen. 1994. Single-stage multiple comparison procedures under heteroscedasticity. American Journal of Mathematical and Management Sciences 14(1,2):1-48.

Google Scholar

[71]

Yang, W.N. and B.L. Nelson. 1989. Optimization using common random numbers, control variates, and multiple comparisons with the best. Proceedings of the 1989 Winter Simulation Conference, 1112-1120.

Digital Library

Google Scholar

[72]

Yang, W.N. and B.L. Nelson. 1991. Using common random numbers and control variates in multiple-comparison procedures. Operations Research 39(4):583-591.

Digital Library

Google Scholar

[73]

Yuan, M. and B.L. Nelson. 1993. Multiple comparisons with the best for steady-state simulation. ACM Transactions on Modeling and Computer Simulation 3(1):66-79.

Digital Library

Google Scholar

[74]

Zinger, A. 1961. Detection of the best and outlying normal populations with known variances. Biometrika 48:457-461.

Crossref

Google Scholar

[75]

Zinger, A. and J. St. Pierre. 1958. On the choice of the best amongst three normal populations with known variances. Biometrika 45:436-446.

Crossref

Google Scholar

Cited By

View all

Finck S(2016)Analysis of simple pattern search on the noisy sphere model2016 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC.2016.7744338(4313-4320)Online publication date: Jul-2016
https://doi.org/10.1109/CEC.2016.7744338
Wolf GAverbuch ANeittaanmäki P(2014)Parameter Rating by Diffusion GradientModeling, Simulation and Optimization for Science and Technology10.1007/978-94-017-9054-3_13(225-248)Online publication date: 19-Jun-2014
https://doi.org/10.1007/978-94-017-9054-3_13
Chang HHu J(2012)On the Probability of Correct Selection in Ordinal Comparison over Dynamic NetworksJournal of Optimization Theory and Applications10.1007/s10957-012-0082-x155:2(594-604)Online publication date: 1-Nov-2012
https://dl.acm.org/doi/10.1007/s10957-012-0082-x
Show More Cited By

Index Terms

A survey of ranking, selection, and multiple comparison procedures for discrete-event simulation
1. Computing methodologies
  1. Modeling and simulation
    1. Model development and analysis
    2. Simulation types and techniques
      1. Discrete-event simulation

Recommendations

Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey

An important use for discrete-event simulation models lies in comparing and contrasting competing design alternatives without incurring any physical costs. This article presents a survey of the literature for two widely used classes of statistical ...
On fidelity and model selection for discrete event simulation

In simulation, perhaps the most common use of the term 'fidelity' refers to the faithfulness with which model behavior reflects modeled system behavior. While there have been studies of fidelity seeking absolute and quantitative measures, there is not ...
Inside discrete-event simulation software: how it works and why it matters
WSC '10: Proceedings of the Winter Simulation Conference

This paper provides simulation practitioners and consumers with a grounding in how discrete-event simulation software works. Topics include discrete-event systems; entities, resources, control elements and operations; simulation runs; entity states; ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

WSC '99: Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future - Volume 1

December 1999

925 pages

ISBN:0780357809

DOI:10.1145/324138

Editors:
Phillip A. Farrington
Univ. of Alabama in Huntsville, Huntsville
,
Harriet Black Nembhard
Univ. of Wisconsin—Madison, Madison
,
David T. Sturrock
Systems Modeling, Sewickley, PA
,
Gerald W. Evans
Univ. of Louisville, Louisville, KY

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 1999

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

WSC99

Sponsor:

WSC99: Winter Simulation Conference 1999

December 5 - 8, 1999

Arizona, Phoenix, USA

Acceptance Rates

WSC '99 Paper Acceptance Rate 139 of 206 submissions, 67%;

Overall Acceptance Rate 3,413 of 5,075 submissions, 67%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
685
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)3

Reflects downloads up to 16 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Finck S(2016)Analysis of simple pattern search on the noisy sphere model2016 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC.2016.7744338(4313-4320)Online publication date: Jul-2016
https://doi.org/10.1109/CEC.2016.7744338
Wolf GAverbuch ANeittaanmäki P(2014)Parameter Rating by Diffusion GradientModeling, Simulation and Optimization for Science and Technology10.1007/978-94-017-9054-3_13(225-248)Online publication date: 19-Jun-2014
https://doi.org/10.1007/978-94-017-9054-3_13
Chang HHu J(2012)On the Probability of Correct Selection in Ordinal Comparison over Dynamic NetworksJournal of Optimization Theory and Applications10.1007/s10957-012-0082-x155:2(594-604)Online publication date: 1-Nov-2012
https://dl.acm.org/doi/10.1007/s10957-012-0082-x
Poropudas JVirtanen KDunkin AIngalls R(2009)Influence diagrams in analysis of discrete event simulation dataWinter Simulation Conference10.5555/1995456.1995561(696-708)Online publication date: 13-Dec-2009
https://dl.acm.org/doi/10.5555/1995456.1995561
Poropudas JVirtanen K(2009)Influence diagrams in analysis of discrete event simulation dataProceedings of the 2009 Winter Simulation Conference (WSC)10.1109/WSC.2009.5429683(696-708)Online publication date: Dec-2009
https://doi.org/10.1109/WSC.2009.5429683
Zhang HTam CLi HShi J(2006)Particle Swarm Optimization-Supported Simulation for Construction OperationsJournal of Construction Engineering and Management10.1061/(ASCE)0733-9364(2006)132:12(1267)132:12(1267-1274)Online publication date: Dec-2006
https://doi.org/10.1061/(ASCE)0733-9364(2006)132:12(1267)
Thompson ACluett W(2005)Stochastic iterative dynamic programmingAutomatica (Journal of IFAC)10.1016/j.automatica.2004.12.00341:5(767-778)Online publication date: 1-May-2005
https://dl.acm.org/doi/10.1016/j.automatica.2004.12.003
Jones MWhite K(2004)Stochastic approximation with simulated annealing as an approach to global discrete-event simulation optimizationProceedings of the 36th conference on Winter simulation10.5555/1161734.1161828(500-507)Online publication date: 5-Dec-2004
https://dl.acm.org/doi/10.5555/1161734.1161828
Jones MWhite K(2004)Stochastic Approximation with Simulated Annealing as an Approach to Global Discrete-Event Simulation OptimizationProceedings of the 2004 Winter Simulation Conference, 2004.10.1109/WSC.2004.1371354(488-495)Online publication date: 2004
https://doi.org/10.1109/WSC.2004.1371354

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey

On fidelity and model selection for discrete event simulation

Inside discrete-event simulation software: how it works and why it matters