Quality measures for embedded systems and their application to control and certification

Distributed, real-time embedded (DRE) systems, such as that being developed under the DARPA Adaptive Reflective Middleware System (ARMS) program, require predictable, controlled real-time behavior. It is a challenge to develop ways to measure and evaluate the quality, or utility, of DRE system performance in the context of the dynamic, unpredictable environments in which they operate. This difficulty runs hand-in-hand with the challenge of providing adequate control of the system to effect behavior in the system that will allow the controlled system to be certified as exhibiting correct behavior. As part of the ARMS program, we have been developing utility-based measures of system quality for assessing correct behavior of a system and for driving feedback control at runtime. This paper describes the real-time utility measures we have developed for ARMS and their use as feedback signals to a control system that manages the dynamic allocation of resources to applications in the system. We then explore the issues associated with using utility-based assessment as part of an evidence-based certification process for dynamic real-time systems.