Experimental Assessment of Accuracy of Automated Knowledge Capture

The U.S. armed services are widely adopting simulation-based training, largely to reduce costs associated with live training. However simulation-based training still requires a high instructor-to-student ratio which is expensive. Intelligent tutoring systems target this need, but they are often associated with high costs for knowledge engineering and implementation. To reduce these costs, we are investigating the use of machine learning to produce models of expert behavior for automated student assessment. A key concern about the expert modeling approach is whether it can provide accurate assessments on complex tasks of real-world interest. This study evaluates of the accuracy of model-based assessments on a complex task. We trained employees at Sandia National Laboratories on a Navy simulator and then compared their simulation performance to the performance of experts using both automated and manual assessment. Results show that automated assessments were comparable to the manual assessments on three metrics.

Authors and Affiliations

1. Sandia National Laboratories, P.O. Box 5800, Albuquerque, NM 87185, USA

   Susan M. Stevens, J. Chris Forsythe, Robert G. Abbott & Charles J. Gieseler

Editors and Affiliations

1. Office of Naval Research, United States Navy, 875 N. Randolph Road, VA 22203, Arlington, USA

   Dylan D. Schmorrow

2. Office of Naval Research, 875 N. Randolph Road, VA 22203, Arlington, USA

   Ivy V. Estabrooke

3. Defense Materiel Organization, Royal Netherlands Navy, P.O. Box 20702, 2500 ES, The Hague, The Netherlands

   Marc Grootjen

Reprints and permissions

© 2009 Springer-Verlag Berlin Heidelberg

Policies and ethics