WO2005043331B1

WO2005043331B1 - Method and apparatus for creating and evaluating strategies

Info

Publication number: WO2005043331B1
Application number: PCT/US2004/036149
Authority: WO
Inventors: Scott Malcolm Caplan; Yen Fook Chang; Michael Raymond Cohen; Stuart Crawford; Favero Brendan Del; Gerald Fahner; Robert Mun-Cheong Fung; Arthur Bruce Hoadley; Jun Hua; Chisoo S Lyons; John Perlis; Nina Shikaloff; Gary Sullivan; Aush Thaker; Eric C Wells
Original assignee: Fair Isaac Corp
Priority date: 2003-10-29
Filing date: 2004-10-29
Publication date: 2005-11-03
Also published as: US20050096950A1; WO2005043331A2; WO2005043331A3

Abstract

A method and apparatus for strategy science methodology involving computer implementation is provided. The invention includes a well-defined set of procedures for carrying out a full range of projects to develop strategies for clients. One embodiment of the invention produces custom consulting projects that are found at one end of the full range of projects. At the other end of the range are, for example, projects developing strategies from syndicated models. The strategies developed are for single decisions or for sequences of multiple decisions. Some parts of the preferred embodiment of the invention are categorized into the following areas: Team Development, Strategy Situation Analysis, Quantifying the Objective Function, Data Request and Reception, Data Transformation and Cleansing, Decision Key and Intermediate Variable Creation, Data Exploration, Decision Model Structuring, Decision Model Quantification, An Exemplary Score Tuner, Strategy Creation, An Exemplary Strategy Optimizer, An Exemplary Uncertainty Estimator, and Strategy Testing. Each of the sub-categories are described and discussed in detail under sections of the same headings. The invention uses judgment in addition to data for developing strategies for clients.

Claims

AMENDED CLAIMS [received by the International Bureau on 07 September 2005 (07.09.05); original claims 155-176 ; On replacement sheets 155-176, Claims 1, 6, 28, 33, 43, 46, 55, 57, 59, 61, 66, 68 and0 are replaced by the amended claims bearing the same numbers; and claims 2-5, 7-27, 29-32, 34-42, 44-45, 47- 54, 56, 58, 60, 62-65, 67, 69 and 71 are unchanged] + Statement ARTICLE 19 REPLACEMENT CLAIMS 1. An iterative method for any of creating and evaluating at least one 5 strategy, comprising the steps of: providing at least two consecutive modules comprising any of: a team development module for developing a strategy modeling team; a strategy situation analysis module for framing a decision situation; a data request and reception module for designing and executing0 logistics of specifying, acquiring, and loading data required for decision and strategy modeling; a data transformation and cleansing module for verifying, cleansing, and transforming data; a decision key and intermediate variable creation module for5 computing additional variables from data and constructing a data dictionary; a data exploration module for determining characteristics that are effective decision keys and intermediate variables; a decision model structuring module for formalizing relationships between decisions, decision keys, intermediate variables, and value of a0 decision model; a decision model quantification module for encoding information into a decision model; a strategy creation module for determining strategies that a client can test; and5 a strategy testing module for testing strategies to guide refinement of strategies and refinement of a decision model and to select a best strategy for deployment; wherein each of the modules has a capability to pass control to the next consecutive module;0 wherein at least one of the modules has a capability to iteratively pass control to a prior consecutive module; and wherein each of said modules has a capability to interact with an expert task manager, wherein said expert task manager provides expert knowledge about any of strategy modeling processes and sub-processes.

2. The iterative method of Claim 1 , the step of providing said team development module further comprising: said strategy modeling team executing analysis to allow a leader of said strategy modeling team to convince a decision maker to implement a strategy favored by said analysis.

3. The iterative method of Claim 1 , the step of providing said strategy situation analysis module further comprising: identifying the values of the organization; and ensuring that the right decisions and strategies are considered in an analysis.

4. The iterative method of Claim 1 , the step of providing said data request and reception module further comprising: designing and executing logistics of specifying, acquiring, and loading data required for decision and strategy modeling.

5. The iterative method of Claim 1 , the step of providing said data transformation and cleansing module further comprising: verifying, cleansing, and transforming data.

6. The iterative method of Claim 1 , the step of providing said decision key and intermediate variable creation module further comprising: computing intermediate variables from said data, said intermediate variables dependent on decision keys; and constructing a data dictionary.

7. The iterative method of Claim 1 , the step of providing said data exploration module further comprising: providing insight into said data by determining which decision keys are most relevant for predicting said intermediate variables; and gaining insight into a customer's business and business processes.

178

8. The iterative method of Claim 1 , the step of providing said decision model structuring module further comprising: formalizing relationships between decisions, decision keys, intermediate variables, and value by connecting such in a model.

9. The iterative method of Claim 1 , the step of providing said decision model quantification module further comprising: encoding information into a decision model.

10. The iterative method of Claim 1 , the step of providing said strategy creation module further comprising: applying optimization methods to a decision model to determine an optimal strategy for a set of cases.

11. The iterative method of Claim 1 , the step of providing said strategy creation module further comprising: evolving using results from a decision model being enriched and from strategies tested.

12. The iterative method of Claim 1 , the step of providing said strategy testing module further comprising: providing means for evaluating each strategy based on simulation; and providing means for evaluating a strategy in the field.

13. The iterative method of Claim 1 , further comprising the steps of: beginning with a simplified value model having less than eight drivers; wherein each of said drivers is modeled crudely by one or two decision keys; initially including no constraints; using said simplified value model for beginning said strategy creation module and said strategy testing module, said strategy creation module and said strategy testing module indicating areas of said decision model where refinement adds particular value; and

179 after interaction between said decision model and strategies is acceptable, iteratively adding details reflecting limitations of a business process.

14. The iterative method of Claim 1 , wherein said team development module comprises a team creation component and a decision quality component.

15. The iterative method of Claim 1 , further comprising the step of: providing a decision quality process for enabling an organization to systematically identify, understand, and track views of quality of decision making.

16. The iterative method of Claim 1 , further comprising the step of: providing any of six dimensions associated with any of six links in a decision quality chain, said any of six links comprising: appropriate frame; creative-feasible alternatives; meaningful-reliable Information; clear values and tradeoffs; logically-correct reasoning; and commitment to action; wherein said chain supports an organization's value.

17. The iterative method of Claim 1 , said step of providing a strategy situation analysis module further comprising the steps of: framing a problem by: identifying issues; developing a decision hierarchy; understanding an organization's values; and brainstorming and clarifying alternatives; further understanding said organization's values by: developing value metrics and prototyping metric results; and planning for data acquisition by: identifying intermediate variables; and

180 developing a plan for assessment; wherein for clarification: optionally returning to said framing a problem step after said further understanding said organization's values step; and optionally returning to said further understanding said organization's values step after said planning for data acquisition step.

18. The iterative method of Claim 1 , the step of providing said data request and reception module further comprising the steps of: developing data parameters, including: determining data elements; designing a performance period; determining data records; and constructing an initial data dictionary; determining transfer parameters, including: determining transfer format; and determining transfer method; preparing data, including: assembling transfer data; and transferring data; and loading data on a target system.

19. The iterative method of Claim 1 , said step of providing a data transformation and cleansing module further comprising the steps of: validating original data sets, comprising: investigating original data sets; and cleaning original data sets; creating analysis data sets, comprising; and transforming data; and computing additional variables; validating analysis data sets, comprising; transforming data; and computing additional variables; wherein while creating analysis data sets and problems are uncovered

181 in original data sets, then original data sets are further cleaned and retransformed; and wherein while validating analysis data sets and problems in said transformation, or in original data sets, are uncovered, then such tasks are revisited.

20. The iterative method of Claim 1 , said step of providing a decision key and intermediate variable creation module further comprising the steps of: first creating dependent variables useful for decision models, comprising: identifying concepts; triaging concepts; and defining dependent variables; and creating independent variables useful for decision models, comprising identifying concepts; triaging concepts; and defining dependent variables; wherein intermediate variables depend on decision keys, other intermediate variables, or decisions; and wherein each intermediate variable encapsulates a predictive model with a dependent variable and independent variables.

21. The iterative method of Claim 1 , said step of providing a data exploration module further comprising the steps of: applying basic statistical analysis, comprising: analyzing continuous variables; and analyzing discrete variables; applying variable reduction techniques, comprising: applying human and business judgment; and applying computational methods; applying advanced statistical analysis; verifying results; and presenting said results.

182

22. The iterative method of Claim 1 , said step of providing a decision model structuring module further comprising the steps of: conceptualizing, comprising the steps of: selecting intermediate variables that drive value; building coarse models of intermediate variables; and verifying constraints; and drawing a decision model structure; wherein said conceptualizing step is iteratively available for use after said drawing step.

23. The iterative method of Claim 1 , said step of providing a decision model quantification module further comprising the steps of: modeling intermediate variables; filling in nodes with models, functions, and/or constants; and validating said decision model; wherein said modeling step is iteratively available from said filling in step, and wherein said filling in step is iteratively available from said validating said decision model step.

24. The iterative method of Claim 1 , further comprising the step of providing a score tuner component for automating decision model updating and reporting, said score tuner component comprising any of: data awareness capability; triggering rules; model history retention; self-guided model development; connection to a decision engine; and execution and analytic audit trails; wherein when a tuning run is triggered, results are reviewed and either accepted and an update is deployed, or rejected.

25. The iterative method of Claim 1 , said step of providing a strategy creation module further comprising the steps of:

183 performing model optimization, comprising: identifying metric variables; determining optimization parameters; and running optimization; analyzing optimization results, comprising viewing optimization results; and performing sensitivity analysis on constraints; and developing strategies, comprising: building strategies; and refining strategies; wherein the performing model optimization step and the analyzing optimization results step are available to be used iteratively from either the analyzing optimization results step or the developing strategies step.

26. The iterative method of Claim 1 , further comprising the step of: providing a non-linear constrained optimization tool for improving test designs and optimizing strategies.

27. The iterative method of Claim 1 , said step of providing a strategy testing module further comprising the steps of: testing strategies, comprising: performing strategy simulation; and performing field testing; evaluating strategies; and performing active data collection; wherein said testing strategies step is available for being used iteratively from said evaluating strategies step.

28. An apparatus for any of iteratively creating and evaluating at least one strategy, comprising: means for providing at least two consecutive modules comprising any of: a team development module for developing a strategy modeling team; a strategy situation analysis module for framing a decision situation; a data request and reception module for designing and executing logistics of specifying, acquiring, and loading data required for decision and strategy modeling; a data transformation and cleansing module for verifying, cleansing, and transforming data; a decision key and intermediate variable creation module for computing additional variables from data and constructing a data dictionary; a data exploration module for determining characteristics that are effective decision keys and intermediate variables; a decision model structuring module for formalizing relationships between decisions, decision keys, intermediate variables, and value of a decision model; a decision model quantification module for encoding information into a decision model; a strategy creation module for determining strategies that a client can test; and a strategy testing module for testing strategies to guide refinement of strategies and refinement of a decision model and to select a best strategy for deployment; wherein each of the modules has a capability to pass control to the next consecutive module; wherein at least one of the modules has a capability to iteratively pass control to a prior consecutive module; and wherein each of said modules has a capability to interact with an expert task manager, wherein said expert task manager provides expert knowledge about any of strategy modeling processes and sub-processes.

29. The apparatus of Claim 28, said team development module further comprising: means for said strategy modeling team executing analysis to allow a leader of said strategy modeling team to convince a decision maker to implement a strategy favored by said analysis.

185

30. The apparatus of Claim 28, said strategy situation analysis module further comprising: means for identifying the values of the organization; and means for ensuring that the right decisions and strategies considered in an analysis.

31. The apparatus of Claim 28, said data request and reception module further comprising: means for designing and executing logistics of specifying, acquiring, and loading data required for decision and strategy modeling.

32. The apparatus of Claim 28, said data transformation and cleansing module comprising: means for verifying, cleansing, and transforming data.

33. The apparatus of Claim 28, said decision key and intermediate variable creation module further comprising: means for computing intermediate variables from said data, said intermediate variables dependent on decision keys; and means for constructing a data dictionary.

34. The apparatus of Claim 28, said data exploration module further comprising: means for providing insight into said data by determining which decision keys are most relevant for predicting said intermediate variables; and means for gaining insight into a customer's business and business processes.

35. The apparatus of Claim 28, further comprising: means for said decision model structuring module formalizing relationships between decisions, decision keys, intermediate variables, and value by connecting such in a model.

186

36. The apparatus of Claim 28, further comprising: means for said decision model quantification module encoding information into a decision model.

37. The apparatus of Claim 28, further comprising: means for said strategy creation module applying optimization methods to a decision model to determine an optimal strategy for a set of cases.

38. The apparatus of Claim 28, further comprising: means for said strategy creation module evolving using results from a decision model being enriched and from strategies tested.

39. The apparatus of Claim 28, further comprising: means for said strategy testing module: providing means for evaluating each strategy based on simulation; and providing means for evaluating a strategy in the field.

40. The apparatus of Claim 28, further comprising: means for beginning with a simplified value model having less than eight drivers wherein each of said drivers is modeled crudely by one or two decision keys; means for initially including no constraints; means for using said simplified value model for beginning said strategy creation module and said strategy testing module, said strategy creation module and said strategy testing module indicating areas of said decision model where refinement adds particular value; and means for after interaction between said decision model and strategies is acceptable, iteratively adding details reflecting limitations of a business process.

187

41. The apparatus of Claim 28, wherein said team development module comprises: a team creation component; and a decision quality component.

42. The apparatus of Claim 28, further comprising: means for providing a decision quality process for enabling an organization to systematically identify, understand, and track views of quality of decision making.

43. The apparatus of Claim 28, further comprising: means for providing any of six dimensions associated with any of six links in a decision quality chain, said six links comprising: appropriate frame; creative-feasible alternatives; meaningful-reliable Information; clear values and tradeoffs; logically-correct reasoning; and commitment to action; wherein said chain supports an organization's value.

44. The apparatus of Claim 28, said means for providing a strategy situation analysis module further comprises: means for framing a problem by: identifying issues; developing a decision hierarchy; understanding an organization's values; and brainstorming and clarifying alternatives; means for further understanding said organization's values by developing value metrics and prototyping metric results; and means for planning for data acquisition by: identifying intermediate variables; and developing a plan for assessment; wherein for clarification:

188 optional means for returning to said framing a problem step after said further understanding said organization's values step; and optional means for returning to said further understanding said organization's values step after said planning for data acquisition step.

45. The apparatus of Claim 28, said data request and reception module further comprising: means for developing data parameters, comprising any of: determining data elements; designing a performance period; determining data records; and constructing an initial data dictionary; means for determining transfer parameters, comprising: determining transfer format; and determining transfer method; means for preparing data, comprising: assembling transfer data; and transferring data; and means for loading data on a target system.

46. The apparatus of Claim 28, said means for providing a data transformation and cleansing module further comprising: means for validating original data sets, comprising: investigating original data sets; and cleaning original data sets; means for creating analysis data sets, comprising: transforming data; and computing additional variables; and means for validating analysis data sets, comprising: transforming data; and computing additional variables; wherein while creating analysis data sets and problems are uncovered in original data sets, then original data sets are further cleaned and retransformed; and

189 wherein while validating analysis data sets and problems in said transformation, or in original data sets, are uncovered, then such tasks are revisited.

47. The apparatus of Claim 28, said means for providing a decision key and intermediate variable creation module further comprising: means for first creating dependent variables useful for decision models, comprising: identifying concepts; triaging concepts; and defining dependent variables; and means for creating independent variables useful for decision models, comprising identifying concepts; triaging concepts; and defining dependent variables; wherein intermediate variables depend on decision keys, other intermediate variables, or decisions; and wherein each intermediate variable encapsulates a predictive model with a dependent variable and independent variables.

48. The apparatus of Claim 28, said means for providing a data exploration module further comprising: means for applying basic statistical analysis, comprising: analyzing continuous variables; and analyzing discrete variables; means for applying variable reduction techniques, comprising: applying human and business judgment; and applying computational methods; means for applying advanced statistical analysis; verifying results; and presenting said results.

190

49. The apparatus of Claim 28, said means for providing a decision model structuring module further comprising: means for conceptualizing, comprising the steps of: selecting intermediate variables that drive value; building coarse models of intermediate variables; and verifying constraints; and means for drawing a decision model structure; wherein said conceptualizing step is iteratively available for use after said drawing step.

50. The apparatus of Claim 28, said means for providing a decision model quantification module further comprising: means for modeling intermediate variables; means for filling in nodes with models, functions, and/or constants; and means for validating said decision model; wherein said modeling step is iteratively available from said filling in step, and wherein said filling in step is iteratively available from said validating said decision model step.

51. The apparatus of Claim 28, further comprising: means for providing a score tuner component for automating decision model updating and reporting, said score tuner component comprising any of: data awareness capability; triggering rules; model history retention; self-guided model development; connection to a decision engine; and execution and analytic audit trails; wherein when a tuning run is triggered, results are reviewed and either accepted and an update is deployed, or rejected.

52. The apparatus of Claim 28, said means for providing a strategy creation module further comprising:

191 means for performing model optimization, comprising: identifying metric variables; determining optimization parameters; and running optimization; means for analyzing optimization results, comprising viewing optimization results; and performing sensitivity analysis on constraints; and means for developing strategies, comprising: building strategies; and refining strategies; wherein the performing model optimization step and the analyzing optimization results step are available to be used iteratively from either the analyzing optimization results step or the developing strategies step.

53. The apparatus of Claim 28, further comprising: a non-linear constrained optimization tool for improving test designs and optimizing strategies.

54. The apparatus of Claim 28, said means for providing a strategy testing module further comprising: testing strategies, comprising: performing strategy simulation; and performing field testing; and evaluating strategies; and performing active data collection; wherein said testing strategies step is available for being used iteratively from said evaluating strategies step.

55. An apparatus for automating any of decision model updating and reporting, comprising: at least one tuning apparatus, comprising any of: data awareness capability; triggering rules; model history retention;

192 self-guided model development; connection to a decision engine; and means for triggering a parameter tuning run execution and analytic audit trails; and means for reviewing results, wherein said results are either accepted and an update is deployed, or rejected.

56. The apparatus of Claim 55, further comprising: means for interacting with a server that handles tuning parameters, and running a scripted model optimization engine for generating new models and evaluation reports; wherein said tuning parameters are any of sample sizes, population definition, and whether tuning is manually initiated or triggered on a set schedule.

57. A decisioning client apparatus, comprising: a decisioning client application processing system for: supplying data associated with a customer to a decision engine; and requesting a decision; wherein said decision engine comprises a score generation module; means for said decision engine, using said score generation module, generating needed transformations of said data and generating at least one score, said at least one score based on at least one score weight of at least one scorecard at a time; means for said decision engine applying pre-specified decision rules and strategies using said data and said transformed data, and at least one score for generating a vector of recommended decision actions; means for said decision engine returning requested data, said transformed data, said at least one score, information about said at least one scorecard, and said recommended actions to said decisioning client application processing system; and means for said decisioning client application processing system

193 optionally implementing said recommended actions, and storing results into a data store.

58. The decisioning client apparatus of Claim 57, further comprising any of: means for said decisioning client application processing system optionally taking additional non-score-based decisions over time; means for said decisioning client application processing system monitoring and recording periodic signals from customers and general environment; means for said decisioning client application processing system gathering data over time about a customer for helping determine one or more outcomes of interest; and an asynchronous process periodically triggering preparation of a matched data set from information about a customer, said information from a predetermined time, wherein said results are appended to a growing store of predictive plus performance data records; and said asynchronous process further comprising means for a score tuner component having a triggering mechanism, using said triggering mechanism for periodically taking said matched data set and producing, if appropriate, score weight updates of at least one active scorecard, wherein said scorecard is installed into said score generation module after a review.

59. A score tuner method, comprising the steps of: providing a score tuning broker module for performing administrative tasks associated with updating of score weights; and providing a score weight engine module for performing activities related to scorecard results and score weights; wherein said score tuning broker module performs the steps of: determining which scorecards are candidates for tuning; checking any operating scorecards are flagged for updates; and at a pre-specified and parameterized time frequency, determining from a rule database which scorecards are up for score weight re-tuning; extracting needed data set sub-population based on rules determining

194 what sampling window and stratification a current scorecard needs; for a scorecard that is a candidate for re-tuning for the current time stamp: requesting generation of a data set to be used for said tuning; and determining what score weight engine project is associated with said scorecard; passing a reference to said data set and a project id to said score weight engine, and requesting metrics of scorecard performance from said score weight engine; and determining whether updated version is better or not; and wherein said score weight engine module performs the steps of: reporting on an existing scorecard's development measures; computing a scorecard's performance measures on a new sample; auditing new predictive data set to ensure that settings are adequate to cover data values encountered in said new data; creating a new scorecard version of said scorecard being tuned; converting raw records in said new predictive data set into coarse classed records needed for building weights; building and scaling score weights of said newly created scorecard given said new predictive data; and archiving said newly built scorecard and its performance measures.

60. The score tuner method of Claim 59, wherein said score weight engine module is script-driven.

61. A score tuner method, comprising the steps of: providing a scorecard comprising data; and performing any of: providing rapid weights tuning for modifying score weights of said scorecard; and providing rapid score alignment for aligning parameters of said scorecard; wherein said underlying structure of said scorecard's data is not different from original implementation definition.

62. The score tuner method of Claim 61 , further comprising any of the steps of: providing a range from a null set of weights to automated and intelligent variable selection, classing, model building, scaling, and evaluation; providing automated validation of newly developed weights for a fixed set of characteristics against a set of previously developed weights on said same characteristic set; and providing automatic re-alignment of a set of scorecards to scale to a previous set of odds.

63. The score tuner method of Claim 61 , further comprising any of the steps of: providing capability of specifying updating or re-scaling of many models at once; and providing capability of specifying a schedule for automatic scorecard updates and scaling, implying integration into current decision support systems.

64. The score tuner method of Claim 61 , further comprising the step of: providing modeling functionality comprising the steps of: importing of existing scorecards from decision support software; auditing for legal values for scorecard characteristics in a new data set; generating summarized data in preparation of the tuning process including: classing of values of data records variables into those expected by the scorecard characteristics; generating all summarization needed to run proprietary algorithms from a newly provided predictive data set and previously

196 summarized results from past tuning runs; and displaying some summary statistics of records encountered; providing specification of expected scaling parameters; running an algorithm to generate new score weights for scorecard characteristics; running evaluation procedures on newly tuned weights; displaying a scorecard and its evaluation results; fitting of log of odds vs. score to determine expected odds by score; adjusting alignment parameters to match user supplied expectation; exporting of said tuned alignment parameters in a format acceptable to decision support software, and while maintaining version control for said scorecard; and providing ability to sequence any of above mentioned steps.

65. The score tuner method of Claim 61 , further comprising the step of: providing reporting and visualization capabilities, comprising summarized views of new score variable and scorecard characteristics; wherein each view includes a comparison of old weights versus new, if applicable, and wherein data is divided by defined bins of scorecard characteristics.

66. A score tuner apparatus, comprising: a database manager component for managing a collection of cases used in analysis, and for providing a bridge to multiple possible input data files and/or database management systems; a data manager component for providing data records to other data analysis components, one case at a time in the event that said data analysis components are processing cases in a sample point loop, for exposing a data dictionary to other components, and for allowing posting variables generated in said data analysis components back to said database manager for future

197 recall; a modeler component for providing score weight re-optimization and for logging odds to score alignment functionality; a report collection component for providing viewing, printing, and limited editing of a standard set of model evaluation reports generated by said modeler; a workflow controller for controlling flow of multiple business components performing a set of actions that are implied by user specifications and eventually fulfilling desired data preparation, analysis, and/or presentation steps; and an intelligence agent for performing background checks on results from user actions and for providing suggestions if a query against its rule base returns a recommended intelligent action to take.

67. The score tuner apparatus of Claim 66, wherein said intelligence agent comprises: means for guiding specification of analytic steps; means for reacting to interactive analytic actions with suggestions, via agents, for possible changes; and means for automating intelligence-assisted decision-making in a sequence of analytic actions.

68. A system for estimating an uncertainty interval around at least one estimate of at least one expected outcome, comprising: a processor associated with a memory; an input device operable to allow entering and transferring input data to said processor; an output device for displaying human readable results of manipulation of said input data; and one or more communications buses between said input device and said processor and said output device and said processor, respectively; wherein said memory stores at least one program for quantifying said uncertainty interval due to variation based on any of case-level variation, model variation, and portfolio composition, said program performing a

198 sequence of instructions which, when executed by said processor, cause the processor to perform the steps of: causing a decision model to encapsulate case-level variation; implementing non-parametric bootstrapping techniques to capture model variation; using analysis of historic data on holdout samples to describe case-level error distributions; and capturing portfolio composition variation as an integral element of said quantifying said uncertainty interval process.

69. The system of Claim 68, wherein said process of quantifying said uncertainty interval comprises two stages: wherein said first stage is repeated for each component model making up said decision model, resulting in estimating all necessary parameters, and wherein said second stage uses said estimated parameters for rolling said up variations into aggregated measures and presenting a range of said at least one expected outcome.

70. A method for estimating an uncertainty interval around at least one estimate of at least one expected outcome, comprising the steps of: providing an input device operable to allow any of entering and transferring input data to a processor associated with a memory; providing an output device for displaying human readable results of manipulation of said input data; providing communications buses between said input device and said processor and said output device and said processor, respectively; and storing at least one program at said memory, the program for quantifying said uncertainty interval due to variation based on any of case- level variation, model variation, and portfolio composition, wherein said program performs a sequence of instructions which, when executed by said processor, cause the processor to perform the steps of: providing a decision model to encapsulate case-level variation; implementing non-parametric bootstrapping techniques to capture model variation;

200 using analysis of historic data on holdout samples to describe case- level error distributions; and capturing portfolio composition variation as an integral element of said quantifying said uncertainty interval process.

71. The method of Claim 70, wherein said process of quantifying said uncertainty interval comprises two stages: wherein said first stage is repeated for each component model making up said decision model, resulting in estimating all necessary parameters, and wherein said second stage uses said estimated parameters for rolling said up variations into aggregated measures and presenting a range of said at least one expected outcome.

201