Exploring Deep Reinforcement Learning for Holistic Smart Building Control

Energy saving in buildings is crucial for society due to the significant energy consumption of buildings, accounting for 32% of global energy consumption and 51% of electricity demand worldwide [42, 48]. Advanced control strategies for operating HVAC systems have the potential to improve energy efficiency and human comfort. Especially, in recent years, with the development of the Internet of Things (IoT) [61, 62] and smart buildings, massive building sensor data are available for intelligent management [43] and behavior analysis [10]. Rule-based control (RBC) is a commonly used method to set actuators in heating, ventilation, and air-conditioning (HVAC) systems, such as temperature and fan speed. However, RBC typically relies on static thresholds or simple control loops based on the experience of engineers and facility managers, which may not be optimal and often require adjustments for new buildings during commissioning. These rules are often updated in an ad-hoc manner, based on feedback from occupants or trial and error by HVAC engineers during building operation. Consequently, model-based approaches have emerged to model the thermal dynamics of buildings and implement advanced control algorithms, such as Proportional Integral Derivative (PID) [50] and Model Predictive Control (MPC) [11], to optimize energy consumption and improve building performance. However, accurately modeling the complex thermal dynamics and accounting for the multitude of influencing factors in buildings can be challenging, leading to simplified models that can handle the data requirements for parameter fitting and computational complexity when solving optimization problems [11]. The intricacies of factors such as weather conditions, occupancy patterns, and building envelope characteristics make it difficult to create precise models. As a result, simplified models are often used, which may not fully capture the complexity of real-world building systems.

To overcome the limitations of model-based methods, alternative model-free approaches based on reinforcement learning (RL) have been proposed for HVAC control, including Q-learning [39] and Deep Reinforcement Learning (DRL) [63]. RL allows for the learning of an optimal control policy through trial-and-error interactions between a control agent and a building, without explicitly modeling the system dynamics. DRL-based approaches leverage deep neural networks as control agents, enabling them to handle large state and action spaces in building control [63]. Recent works [58, 63] have demonstrated that DRL can achieve real-time control for improving building energy efficiency. However, existing methods have typically focused on single subsystems within buildings, such as the HVAC system [58] or the heating system [63], while disregarding other subsystems that can impact energy consumption and user comfort from a holistic perspective.

In recent times, there has been a growing trend towards equipping buildings with automatically-adjustable windows and blinds. These cutting-edge solutions, such as the intelligent products offered by GEZE [2], incorporate motor-operated windows and blinds that can be controlled to optimize natural ventilation strategies. By automatically adjusting the windows and blinds, buildings can effectively regulate the inflow of fresh air and natural light, reducing the reliance on mechanical systems for HVAC. Furthermore, researchers [18] have been actively investigating the potential of joint control strategies that integrate the HVAC system with other building subsystems, such as blinds [52], lighting [14], and windows [57]. For instance, studies have shown that enabling window-based natural ventilation can result in significant energy savings for HVAC systems. According to research findings, the energy consumed by HVAC can be reduced by 17% to 47% by leveraging window-based natural ventilation strategies [57]. This underscores the importance of considering the synergies between different building subsystems and adopting integrated control approaches to achieve optimal energy performance.

In our study, we propose a comprehensive approach that takes into account all available subsystems within buildings, including HVAC, blinds, windows, and lighting, to achieve specific building energy efficiency and human comfort goals. Modern buildings are complex systems comprising multiple interconnected subsystems that work in tandem to ensure human comfort, encompassing thermal comfort, visual comfort, and indoor air quality. As illustrated in Figure 1, various subsystems (HVAC, Blind, Window, and Lighting system) can influence indoor temperature. For instance, the HVAC system plays a crucial role in regulating indoor temperature by adjusting discharge temperature set points at the Variable Air Volume (VAV) level. Manipulating blind slats provides control over the entry of external sunlight, which can impact indoor air temperature. Additionally, regulating the window system enables the exchange of indoor and outdoor air, further influencing indoor temperature. These interconnected subsystems need to work harmoniously to achieve optimal energy efficiency and comfort.

To achieve more efficient energy management in buildings, our proposed study focuses on the joint control problem of four subsystems, as depicted in Figure 2. These subsystems play a crucial role in meeting three human comfort metrics. The energy consumption of a building is influenced by the interactions among these subsystems, making it challenging to control them jointly. This is because they may have opposite effects on different human comfort metrics. For instance, opening a window can improve indoor air quality and save energy consumed by the HVAC system for ventilation. However, it may also result in a reduction of indoor temperature in winter or an increase in summer. To mitigate the temperature variation caused by an open window, the HVAC system may need to expend more energy, which could offset the energy savings from natural ventilation.

This paper introduces OCTOPUS, a customized DRL-based control system designed to optimize energy management in buildings by controlling four subsystems to meet three human comfort requirements with maximum energy efficiency. OCTOPUS leverages the advantages of DRL-based control, including adaptability to new buildings, real-time actuation, and handling of large state spaces. However, there are three main challenges in controlling the four subsystems jointly within a unified framework.

The first challenge is the high-dimensionality of control actions. With a uniform DRL framework, OCTOPUS needs to decide on control actions for four subsystems simultaneously and periodically, including HVAC temperature, electric lights brightness, blind slat range, and window opening proportion. Each subsystem adds one dimension to the action space, resulting in an extremely large set of possible action combinations. To address this, a novel neural architecture is proposed, featuring a shared representation followed by four network branches, one for each action dimension. Additionally, a state value obtained from the shared representation is added to the output of the four branches, allowing for independence in each action dimension while maintaining joint interrelations in the action space.

The second challenge is defining a reward function that captures the trade-offs between energy consumption and human comfort requirements. To tackle this, the optimization problem is formulated as a reward function in the DRL framework. The proposed reward function jointly considers energy consumption, thermal comfort, visual comfort, and indoor air quality, offering better control and flexibility to meet unique user requirements.

The third challenge is the requirement for a large amount of training data. Model-free approaches, including RL techniques, require substantial training data, which may not be readily available from building stakeholders. To overcome this, a calibrated building simulator combined with weather data is used to generate the necessary training data. OCTOPUS is trained with 10 years of weather data from distinct locations, Merced, CA and Chicago, IL, to ensure adaptability to different building types and weather profiles. The main contributions of this paper are highlighted as follows:

In this section, we perform a set of preliminary simulations in EnergyPlus [45] to understand the relationships between the different subsystems and their impact on human comfort in a building, as described in Figure 2. This also helps us gain trust that the simulator is functioning correctly, providing intuitive and understandable results. Our goal is to study the effect of different subsystems on three human comfort metrics. We model a single-floor office building of 100 \(m^2\) in Merced, California, equipped with a north-facing single-panel window of 2 \(m^2\) and an interior blind. The simulations are conducted with weather data for the month of October, which represents a shoulder season with slightly cold outdoor temperatures but mostly sunny days, resulting in high solar gain.

Effects of HVAC, Blind, and Window Systems on Thermal Comfort. Figure 3 illustrates the impact of three subsystems on thermal comfort, evaluated using the Predictive Mean Vote (PMV) index. A PMV value close to zero indicates optimal thermal comfort, with higher positive values indicating higher temperatures (hot) and lower negative values indicating lower temperatures (cold). A detailed explanation of PMV values and ranges can be found in Section 3.4.2. The baseline case (green-solid) represents when all three subsystems are closed, resembling a “fishtank” model where the only impact on the room is from solar gain during the day through the window. When only the blind is open (blue-dashed), the PMV value increases from 1.45 to 1.75, indicating a rise in temperature due to increased solar gain. This effect is most pronounced during the middle of the day when the sun is at its peak. On the other hand, when the window is open (red-dashed-dot), the PMV value decreases due to the temperature effect, as colder outside air enters the room, resulting in a cooler and more comfortable temperature. The HVAC system (black-dot) maintains the PMV value within an acceptable range (-0.5 to +0.5) by regulating the temperature of the air forced through the room vents. Based on the results in Figure 3, it can be concluded that all three subsystems have a significant impact on thermal comfort.

Effects of Blind system on Visual Comfort and Temperature. Figure 4 depicts illuminance measurements near the window area from 5 am to 7 pm with the blind open and natural light present. Illuminance values of 500-1000 lux or higher are acceptable in most environments. The graph reveals that, for the majority of the day, illuminance values remain within this range when the blind is open, emphasizing the positive impact of open blinds in maintaining adequate natural light levels in the room. In Figure 5, the indoor temperature is compared with the blind open (red-dashed) versus closed (blue-solid), while the outdoor temperature (green-dash-dot) remains lower due to the “fish tank” effect and lack of window ventilation or HVAC system operation during the day. Combining the findings from Figures 4 and 5, it is evident that the blind system can reduce the energy consumed by the lighting system by utilizing natural light, but it may also increase the load on the HVAC system for maintaining indoor temperature. However, during colder outdoor temperatures in winter, sunlight passing through the blind can raise the indoor temperature and save energy used by the HVAC system.

The simulations are carried out to demonstrate the intricate interactions between subsystems and their impact on human comfort. Quantifying the complex relationships among different subsystems and the three human comfort metrics poses a challenge, providing motivation for our work.

This section provides a comprehensive description of OCTOPUS’s design, encompassing a system overview, DRL-based building control, branching dueling Q-Network, and reward function calculation.

This section provides a comprehensive overview of the implementation of OCTOPUS, covering various aspects such as platform setup, HVAC modeling and calibration, and OCTOPUS training. Each component is explained in detail below.

In this section, we conduct a performance comparison between OCTOPUS, the rule-based method, and the latest DRL-based method.

Conventional control of the HVAC system. Model predictive control (MPC) models have been developed for HVAC control. It is a planning-based method that solves an optimal control problem iteratively over a receding time horizon. Some of the advantages of MPC are that it takes into consideration future disturbances and that it can handle multiple constraints and objectives, e.g., energy consumption and human comfort [11]. However, it can be argued that the main roadblock preventing the widespread adoption of MPC is its reliance on a model [36, 47]. By some estimates, modeling can account for up to 75% of the time and resources required for implementing MPC in practice [49]. Because buildings are highly heterogeneous, a custom model is required for each thermal zone or building under control.

There are two paradigms for modeling building dynamics: physics-based and statistics-based [47]. Physics-based models, e.g., EnergyPlus, utilize physical knowledge and material properties of a building to create detailed representation of the building dynamics. A major shortcoming is that such models are not control-oriented. Nonetheless, it is not impossible to use such models for control [8]. For instance, exhaustive search optimization is used to derive control policy for an EnergyPlus model [64]. Furthermore, a physics-based model requires significant modeling effort, because they have a large number of free parameters to be specified by engineers (e.g., 2,500 parameters for a medium-sized building [31]); and information required for determining these parameters are scattered in different design documents [27].

Statistical models assume a parametric model form, which may or may not have physical underpinnings, and identify model parameters directly from data. Dinh and Kim [20] propose a hybrid control that combines MPC and direct imitation learning to reduce energy cost while maintaining a comfortable indoor temperature. While this approach is potentially scalable, a practical problem is that the experimental conditions required for accurate identification of building systems fall outside of normal building operations [4].

Integration of occupant-driven approaches in building control. In recent years, the delicate balance between energy efficiency and occupant comfort in commercial buildings has been a focal point in building control strategies. While conventional control methods, such as MPC, have shown promise, they often face challenges, particularly in the extensive modeling efforts required for heterogeneous buildings.

The works [9, 34, 35] address key aspects of occupant-building interaction and demand response optimization in microgrids. Baldi et al. [9] propose a switched self-tuning approach to automate occupant-building interaction via smart zoning of thermostatic loads, dynamically regulating set points based on occupancy patterns. Korkas et al. [35] focus on occupancy-based demand response and thermal comfort optimization in microgrids with renewable energy sources, presenting a novel control algorithm for integrating energy generation, consumption, and occupant behavior. Additionally, Korkas et al. [34] contribute to the field by developing a distributed demand management system (D-DMS) for grid-connected microgrids, incorporating feedback actions for improved performance under changing conditions.

These papers provide valuable insights into dynamically managing occupant comfort while optimizing energy efficiency, addressing limitations in conventional control methods. While our work leverages physics-based and statistical models for building dynamics, the incorporation of feedback-driven strategies from these studies could offer further enhancements in real-world building control systems.

Conventional control of multiple subsystems. Blind system should be considered as an integral part of fenestration system design for commercial and office buildings, in order to balance daylighting requirements versus the need to reduce solar gains. The impact of glazing area, shading device properties and shading control on building cooling and lighting demand was calculated using a coupled lighting and thermal simulation module [52]. The interactions between cooling and lighting energy use in perimeter spaces were evaluated as a function of window-to-wall ratio and shading parameters.

The impacts of window operation on building performance was investigated [57] for different types of ventilation systems including natural ventilation, mixed-mode ventilation, and conventional VAV systems in a medium-size reference office building. While the results highlighted the impacts of window operation on energy use and comfort and identified HVAC energy savings with mixed-mode ventilation during summer for various climates, the control for window opening fraction was estimated by experience and is not salable for different kinds of buildings. Kolokotsa et al. [33] propose an energy-efficient fuzzy controller that utilizes a genetic algorithm to manage four subsystems (HVAC, lighting, window, and blind) and ensure optimal human comfort. However, the drawback lies in the genetic algorithm’s time-consuming nature, taking several minutes to hours to generate a single control action, making it impractical for real-time implementation in building control systems.

RL-based control of the HVAC system. As a general artificial intelligence technology, deep learning [41] and deep reinforcement learning [17] have been applied in many fields, e.g., Internet of energy [40], unmanned aerial vehicles [56], smart microgrids [38], edge computing [13]. Many works also apply RL for HVAC control [44]. RL control can be a “model-free”control method, i.e., an RL agent has no prior knowledge about the controlled process. RL learns an optimal control strategy by “trial-and-error”. Therefore, it can be an online learning method that learns an optimal control strategy during actual building operations. Fazenda et al. [24] investigated the application of a reinforcement-learning-based supervisory control approach, which actively learns how to appropriately schedule thermostat temperature setpoints. However, in HVAC control, online learning may introduce unstable and poor control actions at the initial stage of the learning. In addition, it may take a long time (e.g., over 50 days reported in [24]) for an RL agent to converge to a stable control policy for some cases. Therefore, some studies choose to use an HVAC simulator to train the RL agent offline [63].

Unlike MPC, simulators with arbitrary high complexity can be directly used to train RL agents because of its “model-free” nature. Li and Xia [39] employ Q-learning to regulate HVAC control, while Dalamagkidis et al. [16] introduce a Linear Reinforcement Learning Controller (LRLC) that utilizes linear function approximation of the state-action value function to achieve thermal comfort while minimizing energy consumption. However, tabular Q-learning approaches are not well-suited for problems characterized by a large state space, such as the one involving the four subsystems. Le et al. [37, 54] propose a control method of air free-cooled data centers in tropics via DRL. Vazquez-Canteli et al. [55] develop a multi-agent RL implementation for load shaping of grid-interactive connected buildings. Ding et al. [19] design a model-based RL method for multi-zone building control. Zhang and Lam [63] implement and deploy a DRL-based control method for radiant heating systems in a real-life office building. An et al. [6, 7] design a safe MBRL HVAC control approach that can achieve low human comfort violation with a dynamics model trained on a small dataset. Gao et al. [25] propose deep deterministic policy gradients (DDPGs)-based approach for learning the thermal comfort control policy. Although the above works can improve the performance of HVAC control, they only focus on the HVAC subsystem.

Deploying in a Real Building. Although we have developed a calibrated simulation model of a real building on our campus for training and evaluation purposes, we have not yet deployed OCTOPUS in an actual building due to the lack of access to an automatic blind and window system. However, we are actively seeking financial support to collaborate with our facility team for a potential upgrade that would enable the deployment of OCTOPUS. Our ultimate goal is to deploy OCTOPUS in real buildings, as it has been specifically designed for such applications. For a new building, the deployment process involves creating an EnergyPlus model and calibrating it using real building operation data. Once the OCTOPUS control agent is trained using the calibrated simulation model and real weather data, it can be deployed in the building for real-time control. The control agent takes the building’s state as input and generates control actions for the four subsystems at a predefined action interval (e.g., every 10 minutes). OCTOPUS is capable of providing real-time control, as a single inference takes only 22 ms. In our future work, we plan to deploy OCTOPUS in a real building, where it can demonstrate its effectiveness in achieving energy savings while maintaining human comfort. This practical deployment will allow us to validate the performance and benefits of OCTOPUS in a real-world setting.

Scalability of OCTOPUS. OCTOPUS is currently designed to operate in a one-zone building configuration with a single HVAC system, lighting zone, blind, and window. However, real-world buildings, including small homes, typically consist of multiple lighting zones, blinds, and windows, each requiring individual control actions within a subsystem. Scaling OCTOPUS to handle such complex building configurations is a challenge we aim to address in our future work. To overcome the scalability problem, we plan to extend OCTOPUS by increasing the number of BDQ branches. Each branch will correspond to a subsystem in each zone of the building. By incorporating multiple branches, OCTOPUS will be able to effectively manage and control various subsystems in a building with multiple lighting zones, blinds, and windows. Tackling this scalability issue is crucial to ensure the applicability of OCTOPUS in realistic building environments, where the complexity and diversity of subsystems necessitate advanced control strategies. Our ongoing research aims to develop innovative approaches that enable OCTOPUS to handle large-scale building configurations while maintaining its efficiency and effectiveness in achieving energy savings and optimizing human comfort.

Building Model Calibration. A crucial element of our architecture relies on utilizing a calibrated building model that closely resembles the target building. This allows us to generate the necessary training data effectively. It is not trivial to have a building model in the EnergyPlus simulator. Achieving an accurate calibrated model is indeed a laborious process that involves extensive trial-and-error experimentation with numerous parameters. Among the vast array of parameters available in EnergyPlus, we leveraged our expertise and sought guidance from experts to identify the most critical parameters and determine a reasonable range of values to explore. It took us approximately four weeks to fine-tune the model to an acceptable level. We want to emphasize that this timeframe reflects the complexity of the calibration process and the dedication required for its successful completion.

Despite our efforts, there is no universal solution, and challenges may arise, particularly when dealing with unconventional building architectures or specialized HVAC systems that cannot be readily replicated in a simulation environment. It is not trivial that this model is well-calibrated. These unique cases may require additional considerations and careful adaptation of the simulation model to ensure accurate representation. We acknowledge that achieving a precise calibration can be demanding and time-consuming, but it is a necessary step to ensure the fidelity and reliability of our training process.

Furthermore, it is not trivial to have 10-year data to make sure the model is well-calibrated. We acknowledge the significance of long-term data in validating our calibrated model. In our study, we recognize the importance of extending our analysis over a 10-year period to ensure the robustness of our findings. This extended duration allows us to capture variations in building performance under diverse conditions, contributing to the overall reliability of our approach.

Accepting Users’ Feedback. Certain existing work [60] has introduced a mechanism for users to provide feedback to the control server. This feedback serves as personalized preferences regarding various human comfort metrics and is taken into account during the control decision-making process. OCTOPUS is designed to readily accommodate user feedback, enabling the training of an enhanced agent model with a minor modificationadjusting the calculated comfort values in the reward function based on user feedback. This functionality can be leveraged during the initial training phase or for subsequent updates once the system is deployed. To illustrate, the OCTOPUS control agent can undergo incremental training at specific intervals, such as one month. The newly-trained agent can then be employed for real-time control, incorporating the insights gained from user feedback. This iterative approach allows OCTOPUS to continuously adapt and improve its control strategies based on user preferences and evolving comfort requirements. By fostering this feedback loop, OCTOPUS ensures a more personalized and user-centric control experience in real-world building environments.

The Markov Property in Smart Building Control The Markov Decision Process (MDP) and its inherent Markov property serve as the foundational framework for our study in smart building control. The Markov property, characterized by the independence of future states from past states, plays a pivotal role in shaping the decision-making dynamics within our DRL model. In the context of smart building control, the Markov property simplifies complex decision-making by allowing the DRL agent to focus exclusively on the current state of the environment. This immediate-state dependency aligns seamlessly with the nature of control decisions in smart buildings. For instance, when managing HVAC systems or lighting, actions are primarily influenced by real-time sensor data and environmental conditions, rendering the Markov property a natural fit. By assuming that future states depend only on the present state and action, we streamline the decision-making process, making it computationally tractable for real-time applications.

However, it’s important to recognize that this simplification does come with challenges. Building an accurate Markov model for smart building control requires meticulous calibration, typically involving extensive experimentation with simulation models like EnergyPlus. Achieving a precise match between the model and the real-world building dynamics is laborious but crucial for reliable decision-making. In our future work, we aspire to augment the validation of the Markov property by incorporating empirical data to support the assumption of state independence.

This paper introduces OCTOPUS, an innovative control system for buildings that leverages deep reinforcement learning (DRL) to effectively manage multiple subsystems (e.g., HVAC, lighting, blinds, windows) while balancing energy consumption and human comfort. Our architecture addresses several challenges, including handling large action states, designing a unique reward function that considers both energy efficiency and comfort, and meeting data requirements by utilizing historical weather data and a calibrated simulator specific to the target building.

To evaluate OCTOPUS, we compare its performance against state-of-the-art approaches. We benchmarked it against a rule-based control scheme implemented in a LEED Gold-certified building, as well as a DRL scheme focused on optimized heating from existing literature. The results demonstrate significant energy savings, with OCTOPUS achieving a remarkable 14.26% and 8.1% reduction in energy consumption while consistently maintaining, and sometimes even enhancing, human comfort across temperature, air quality, and lighting aspects. By combining advanced DRL techniques, a tailored reward function, and comprehensive simulations, OCTOPUS showcases its effectiveness in optimizing building control operations. It offers a promising solution for enhancing energy efficiency and occupant comfort in modern buildings, contributing to sustainable and comfortable living environments.