More Web Proxy on the site http://driver.im/

research-article

A Framework for Selecting a Selection Procedure

Authors:

Peter I. Frazier,

Shane G. HendersonAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation (TOMACS), Volume 22, Issue 3

Article No.: 16, Pages 1 - 23

https://doi.org/10.1145/2331140.2331144

Published: 01 August 2012 Publication History

Abstract

For many discrete simulation optimization applications, it is often difficult to decide which Ranking and Selection (R&S) procedure to use. To efficiently compare R&S procedures, we present a three-layer performance evaluation process. We show that the two most popular performance formulations, namely the Bayesian formulation and the indifference zone formulation, have a common representation analogous to convex risk measures used in mathematical finance. We then specify how a decision maker can impose a performance requirement on R&S procedures that is more adequate for her risk attitude than the indifference zone or the Bayesian performance requirements. Such a performance requirement partitions the space of R&S procedures into acceptable and nonacceptable procedures. The minimal computational budget required for a procedure to become acceptable introduces an easy-to-interpret preference order on the set of R&S policies. We demonstrate with a numerical example how the introduced framework can be used to guide the choice of selection procedure in practice.

References

[1]

Allen, T. T. and Bernshteyn, M. 2006. Optimal voting machine analysis. Tech. rep., DRE Analysis for May 2006 Primary, Steven Hertzberg Ed., Cuyahoga County, Technical Report, Election Science Institute.

[2]

Allen, T. T. and Bernshteyn, M. 2008. Helping Franklin County vote in 2008: Waiting lines. Report to the Franklin County Board of Elections.

[3]

Artzner, P., Delbaen, F., Eber, J. M., and Heath, D. 1999. Coherent measures of risk. Math. Finance 9, 3, 203--228.

[4]

Bechhofer, R. E. 1954. A single-sample multiple decision procedure for ranking means of normal populations with known variances. Ann. Math. Statist. 25, 1, 16--39.

[5]

Bechhofer, R. E., Santner, T. J., and Goldsman, D. M. 1995. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. Probability and Statistics. Series, Wiley.

[6]

Berger, J. O. 1985. Statistical Decision Theory and Bayesian Analysis. Springer.

[7]

Boesel, J., Nelson, B. L., and Kim, S. H. 2003. Using ranking and selection to “clean up” after simulation optimization. Oper. Res. 51, 5, 814--825.

Digital Library

[8]

Branke, J., Chick, S. E., and Schmidt, C. 2007. Selecting a selection procedure. Manag. Sci. 53, 12, 1916--1932.

Digital Library

[9]

Chen, C. H. 1996. A lower bound for the correct subset-selection probability and its application to discrete-event system simulations. IEEE Trans. Autom. Control 41, 8, 1227--1231.

[10]

Chen, C. H. and Lee, L. H. 2010. Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. World Scientific Publishing Co. Pte. Ltd., Singapore.

Digital Library

[11]

Chen, C. H., Lin, J., Yücesan, E., and Chick, S. E. 2000. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discr. Event Dynam. Syst. Theory Appl. 10, 3, 251--270.

Digital Library

[12]

Chick, S. E. 1997. Selecting the best system: A decision-theoretic approach. In Proceedings of the Winter Simulation Conference. S. Andradottir, K. J. Healy, D. H. Withers, and B. L. Nelson Eds., IEEE, 326--333.

Digital Library

[13]

Chick, S. E. and Frazier, P. I. 2009. The conjunction of the knowledge gradient and the economic approach to simulation selection. In Proceedings of the Winter Simulation Conference. M. D. Rossetti, R. R. Hill, B. Johansson, A. Dunkin, and R. G. Ingalls Eds., IEEE, 528--539.

Digital Library

[14]

Chick, S. E. and Gans, N. 2009. Economic analysis of simulation selection problems. Manag. Sci. 55, 3, 421--437.

Digital Library

[15]

Chick, S. E. and Inoue, K. 2001a. New procedures to select the best simulated system using common random numbers. Manag. Sci. 47, 8, 1133--1149.

Digital Library

[16]

Chick, S. E. and Inoue, K. 2001b. New two-stage and sequential procedures for selecting the best simulated system. Oper. Res. 49, 5, 732--743.

Digital Library

[17]

DeGroot, M. H. 1970. Optimal Statistical Decisions. McGraw-Hill, New York.

[18]

Föllmer, H. and Schied, A. 2002. Convex measures of risk and trading constraints. Finance Stochast. 6, 4, 429--447.

[19]

Föllmer, H. and Schied, A. 2004. Stochastic Finance, An Introduction in Discrete Time 2nd Ed. Walter de Gruyter & Co., Berlin.

[20]

Hong, J. L. and Nelson, B. L. 2005. The tradeoff between sampling and switching: New sequential procedures for indifference-zone selection. IIE Trans. 37, 7, 623--634.

[21]

Kim, S. H. and Nelson, B. L. 2001. A fully sequential procedure for indifference-zone selection in simulation. ACM Trans. Model. Comput. Simul. 11, 3, 251--273.

Digital Library

[22]

Kim, S. H. and Nelson, B. L. 2006a. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Oper. Res. 54, 3, 475--488.

Digital Library

[23]

Kim, S. H. and Nelson, B. L. 2006b. Selecting the best system. In Handbook in Operations Research and Management Science: Simulation, S. G. Henderson and B. L. Nelson Eds., Elsevier.

[24]

Markowitz, H. M. 1952. Portfolio selection. J. Finance 7, 1, 77--91.

[25]

McNeil, A. J., Frey, R., and Embrechts, P. 2005. Quantitative Risk Management: Concepts, Techniques, and Tools. Princeton University Press, Princeton, NJ.

[26]

Nelson, B. L. and Banerjee, S. 2001. Selecting a good system: Procedures and inference. IIE Trans. 33, 3, 149--166.

[27]

Nelson, B. L. and Matejcik, F. J. 1995. Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Manag. Sci. 41, 12, 1935--1945.

Digital Library

[28]

Paulson, E. 1964. A sequential procedure for selecting the population with the largest mean from k normal populations. Ann. Math. Statist. 35, 1, 174--180.

[29]

Rinott, Y. 1978. On two-stage selection procedures and related probability-inequalities. Comm. Statist. Theory Methods 7, 8, 799--811.

[30]

Santner, T. J. and Tamhane, A. C. 1984. Design of Experiments: Ranking and Selection. Marcel Dekker, Inc.

[31]

Sobel, M. and Huyett, M. J. 1957. Selecting the best one of several binomial populations. Bell Syst. Techn. J. 36, 537--576.

[32]

Swisher, J. R., Jacobson, S. H., and Yücesan, E. 2003. Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey. ACM Trans. Model. Comput. Simul. 13, 2, 134--154.

Digital Library

[33]

Waeber, R., Frazier, P. I., and Henderson, S. G. 2010. Performance measures for ranking and selection procedures. In Proceedings of the Winter Simulation Conference. B. Johansson, S. Jain, J. Montoya-Torres, J. Hugan, and E. Yücesan Eds., IEEE, 1235--1245.

Digital Library

Cited By

Karishma Rao S(2022)Cooperative solutions to exploration tasks under speed and budget constraintsJournal of Simulation10.1080/17477778.2022.204379217:6(676-687)Online publication date: 15-Mar-2022
https://doi.org/10.1080/17477778.2022.2043792
Brandtner PHelfert MAuinger AGaubinger K(2015)Multi-criteria Selection in Design Science Projects --- A Procedure for Selecting Foresight Methods at the Front End of InnovationProceedings of the 10th International Conference on New Horizons in Design Science: Broadening the Research Agenda - Volume 907310.1007/978-3-319-18714-3_19(295-310)Online publication date: 20-May-2015
https://dl.acm.org/doi/10.1007/978-3-319-18714-3_19
Waeber RFrazier PHenderson S(2011)A Bayesian approach to stochastic root findingProceedings of the Winter Simulation Conference10.5555/2431518.2432000(4038-4050)Online publication date: 11-Dec-2011
https://dl.acm.org/doi/10.5555/2431518.2432000

Index Terms

A Framework for Selecting a Selection Procedure
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation theory
2. Mathematics of computing
  1. Mathematical software

Recommendations

A multi-objective selection procedure of determining a Pareto set

Ranking and selection (R&S) procedures have been widely studied and applied in determining the required sample size (i.e., the number of replications or batches) for selecting the best system or a subset containing the best system from a set of k ...
A Fully Sequential Elimination Procedure for Indifference-Zone Ranking and Selection with Tight Bounds on Probability of Correct Selection

<P>We consider the indifference-zone (IZ) formulation of the ranking and selection problem with independent normal samples. In this problem, we must use stochastic simulation to select the best among several noisy simulated systems, with a statistical ...
A Fully Sequential Elimination Procedure for Indifference-Zone Ranking and Selection with Tight Bounds on Probability of Correct Selection

We consider the indifference-zone IZ formulation of the ranking and selection problem with independent normal samples. In this problem, we must use stochastic simulation to select the best among several noisy simulated systems, with a statistical ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 22, Issue 3

August 2012

115 pages

ISSN:1049-3301

EISSN:1558-1195

DOI:10.1145/2331140

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 August 2012

Accepted: 01 May 2012

Revised: 01 March 2012

Received: 01 April 2011

Published in TOMACS Volume 22, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
278
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Karishma Rao S(2022)Cooperative solutions to exploration tasks under speed and budget constraintsJournal of Simulation10.1080/17477778.2022.204379217:6(676-687)Online publication date: 15-Mar-2022
https://doi.org/10.1080/17477778.2022.2043792
Brandtner PHelfert MAuinger AGaubinger K(2015)Multi-criteria Selection in Design Science Projects --- A Procedure for Selecting Foresight Methods at the Front End of InnovationProceedings of the 10th International Conference on New Horizons in Design Science: Broadening the Research Agenda - Volume 907310.1007/978-3-319-18714-3_19(295-310)Online publication date: 20-May-2015
https://dl.acm.org/doi/10.1007/978-3-319-18714-3_19
Waeber RFrazier PHenderson S(2011)A Bayesian approach to stochastic root findingProceedings of the Winter Simulation Conference10.5555/2431518.2432000(4038-4050)Online publication date: 11-Dec-2011
https://dl.acm.org/doi/10.5555/2431518.2432000

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents