WHYNET: a framework for in-situ evaluation of heterogeneous mobile wireless systems

The design and implementation of wireless systems has been impeded by the lack of an evaluation framework that can provide an accurate understanding of middleware and application performance in the context of their interactions with system hardware and software, network architecture and configuration and wireless channel effects. In this paper we present a novel evaluation paradigm wherein the applications, middleware or sub-networks can be evaluated in-situ, in other words, as operational software that interfaces with the operating system and other applications, thus offering a fidelity equivalent to physical deployment. The physical environment in which such systems operate is modeled using high-fidelity simulations. This approach combines the fidelity of physical test beds with the benefits of scalability, repeatability of input parameters, and comprehensive parameter space evaluation - the known limitations of a physical test-bed. The framework design is extensible in that it allows configuring the desired components of a system with different modalities to suit a particular evaluation criterion. The implementation also addresses the key challenges in the interaction of the framework sub-components: seamless interfaces, time synchronization and preserving causality constraints. The benefits and applicability of the framework to diverse wireless contexts is demonstrated by means of various case studies in diverse wireless networks. In one case study, we show that a design exhibiting 4X improvement in network metrics may be actually degrading the application metric by 50%.