Prototyping a Digital Twin System for Environmental Education

The global population is projected to reach 9.7 billion people until 2050 and alongside that exponential growth, the urban population is expected to reach 5.45 billion people (56.2%) [8]. Considering that up to two thirds of our total global population will live in an urban environment by the first half of the 21st century, it is important to consider the impact of this rapid urbanization development on our environment. Research has shown that urbanization is impacting negatively surrounding natural environment by causing changes in air quality, pollution of water, destruction of natural habitats as well as erosion phenomena [24]. In view of the above, our relationship with the natural world becomes important since the status of our global habitat directly affects our own well-being and survival on our planet. Our childhood years are the formative period in which we develop our relationship with the natural world around us and probably the most critical period for forming our future environmental ideals. In this paper we are proposing a novel system for environmental education which aims to encourage children's interaction with the natural world from their school environment, using core technologies of the 4^th Industrial Revolution, namely the technologies of the Digital Twin and Internet of Things (IoT).

The 4^th Industrial Revolution is a technological wave that began from the initiative of the german government to combat the manufacturing challenges of the 21st century [23]. Its core idea is the use of cyberphysical systems that communicate with each other and provide "smart" information about their environment. Core pillar technologies of the 4^th Industrial Revolution are the Internet of Things (IoT), Big Data, Digital Twins as well as cloud computing. The IoT is a network of "smart" devices that communicate with each other and trade information about their own status as well as their environment. The "smart" aspect of these devices refers to their ability to communicate between them through the internet and contribute to context-related decisions based on the information they exchanged about themselves or their environment.

One of the other pillar technologies of the 4^th Industrial Revolution, the Digital Twin, is used mainly today for controlling and monitoring factory processes of production. The idea of the Digital Twin was presented in 2003 by Dr. Grieves in one of his lectures about Product Lifecycle Management (PLM) [7, 18]. The Digital Twin is defined as a digital representation that acts as the digital part of a physical system and is handling each instance of it in real time for the total lifetime of the whole system [9]. So, in essence, the Digital Twin is the digital dimension of a system with 2 parts, a physical and a digital part, which communicate bidirectionally throughout the lifetime of the whole system exchanging information about its functioning and status.

The system we propose in this paper will use the technologies of the Internet of Things and the Digital Twin for transmitting real time information about the natural environment such as atmospheric parameters as well as sounds and photos about bird inhabitants. The system we propose will consist of a smart birdhouse that will collect atmospheric data from its inside, like temperature and humidity, as well as record timestamp photos and sounds when a bird's presence is detected inside. At the same time any collected data will be transmitted to a central server for storing as well as processing before they become available for viewing in a statistics dashboard and a physical-electronic Digital Twin device. The Digital Twin device of the smart birdhouse will play back the collected photos and sounds from the smart birdhouse while the dashboard will show statistics about atmospheric data collected as well as number of times a bird presence was detected inside. This system aims to encourage engagement of children is school environments with the natural world, especially in urban environments where they have few opportunities for such interactions.

In the next sections we will talk about the relationship of children with the natural world as well as the role of environmental education. Next, we will present systems that function in a similar way to our proposed system, as well as show the architecture of our own proposed system, the core technologies that were used to create it and its functioning in real time. In the end of this paper we will discuss about the evaluation of the proposed system and possible future research directions for it.

In this section we will present the logical architecture of the proposed system and introduce the technologies it is based on. Figure 1 presents conceptually the relevant logical architecture.

An architectural block diagram depicting connections between a single outdoor unit of the smart birdhouse, the server used for data collection, multiple instances of the birdhouse digital twins positioned in school classrooms, and a large display terminal or a mobile smartphone terminal for data display.

The system we propose will consist of four basic logical components, the Smart Birdhouse, a dedicated server, a digital dashboard and a digital-electronic Digital Twin of the smart birdhouse. The Smart Birdhouse will be a classical outdoor wooden birdhouse, equipped with appropriate sensors that will collect environmental parameter values from its interior like temperature, humidity and ambient light level, as well as audio or photos from potential bird visitors. The server will be responsible for saving the collected data that is received from the Smart Birdhouse for a certain period of time as well as feed this data to a dashboard platform and a digital electronic twin device. The dashboard platform will show statistical information about the atmospheric data collected like historical values of humidity or temperature and also historic data of times that a bird was detected inside the smart birdhouse, affording interaction with historic data displays and analyses, which can prompt activities based on scientific observation with young learners. Further, the Digital Twin will be a wooden birdhouse replica of the outdoor component, housing an LCD screen and speakers to play back the photos and sounds at the time that they are received from the Smart Birdhouse. This play back will happen very close to the real time events at the smart birdhouse and can act as a near live time observation tool of its interior. Therefore, the Digital Twin will serve as a prompt for continuous engagement and active discussions with nature, at any time during the school curriculum.

The communication between the Smart Birdhouse and the dedicated server that saves the collected data is being carried out through the use of the MQTT protocol [10]. The MQTT protocol is a publish/subscribe communication protocol that is mainly ideal for networks that function at the edge of the internet and usually have a low bitrate of communication ("constrained networks"). The data that is sent from the smart birdhouse to the dedicated server for storing is then handled by the Node-Red platform that runs there, which makes sure to convert the image and sound data in base64 format as well as send all the data in total to the ThingsBoard platform for storing through the use of http requests. These actions are performed by the use of special nodes that are available in the Node-Red platform and can be connected together to create a logical flow of actions that need to be executed like described above. Succinctly the Node-Red platform is a programming tool that gives us the ability to connect devices with API's and other internet services through the use of special nodes.

After the collected data is handled by the Node Red platform then they are sent to the ThingsBoard analytics platform for storage for a certain period of time based on a certain TTL (Time To Live) parameter and also update a dashboard offered by the same platform on statistical values of atmospheric data that was collected. The ThingsBoard platform is an open source platform that can handle the collection, storing and visualization of data that were collected from Internet of Things devices and was chosen due to its versatility in those actions in relation to our system.

In parallel to sending the data to the ThingsBoard platform, photos and sound data collected are sent to an electronic Digital Twin that is equipped with an LCD screen and relevant speakers that provide the ability for media playback.

In this section we will provide an in depth description of how the system of the Smart Birdhouse works, namely we will present the hardware architecture of both the smart birdhouse device as well as its Digital Twin and the way data are sent to and from these two devices through the use of relevant software.

The Smart Birdhouse device is a wooden birdhouse for outdoor use (Fig. 2a), that is equipped with sensors for the collection of atmospheric values in its interior like temperature, humidity and level of ambient light as well as photos/sound from potential bird visitors as we mentioned in the previous section. The electronics that make this data collection possible are housed in an upper compartment of the birdhouse, specially constructed to house these electronics and keep them safe from the bird visitors or potential environmental hazards like rain or wind (Fig. 2c).

Four photographs showing the actual construction of the smart birdhouse from the outside and inside.

A Raspberry Pi 4B microcontroller is used as the computing platform connecting all sensors in the outdoor unit, and responsible for aggregating and transmitting data to the server. We used a variety of sensors like a temperature/humidity sensors (DHT-11), an ambient light sensor (TSL2581), a motion sensor that detects movement based on Passive Infrared Activity (HC-SR501) as well as two microphones, one for recording bird sound with high quality (MEMS microphone) and another one for helping in the detection of bird sound as we will describe later (Mini USB Microphone). A Raspberry Pi camera module is also used to take photos from the inside of the birdhouse when motion is detected. The version of the novel smart birdhouse we developed is not autonomous and needs a USB power cable to power the raspberry pi and the relevant sensors, though the smart birdhouse can become autonomous in the future through the use of solar panels as well as batteries.

4.2 Sound Classification with Edge Impulse

In the smart birdhouse we wanted to record bird sounds when bird presence is detected inside of it as they happen in real time. For this purpose we selected to use the Edge Impulse platform which is a leading solution for machine learning in the edge of a network as well as offering a big range of options and customizability of machine learning models for a variety of classification tasks [16]. In our case we used the Edge Impulse platform for the construction of a machine learning model that would be able to classify sound in one of four potential classes. Three of these sound classes belong to bird species that can have access to the smart birdhouse due to its entrance hole size. Since the entrance hole of the birdhouse we used is 28mm, the bird species that can potentially access it in our testing area of [anonymised for review] were House Sparrow (Passer domesticus), Great Tit (Parus major) and Greenfinch (Chloris chloris). The fourth sound class is noise, which includes a variety of general noise samples like people speaking and other non related noise to our three bird sound classes. With the above considered we made a custom machine learning model in the Edge Impulse platform that can classify received sound from the MEMS microphone to one of the three classes that belong to a bird's sound, or the noise class if no bird sound is recognised. For training we used a Python script to download relevant sample audio data from each of the aforementioned bird species from the Xeno-Canto dataset [22] which constitutes a big sound sample databank of many bird species all around the world supported through a worldwide community of professional and amateur uploaders of sound data. As noise samples for the noise class we used the Google Speech Command Dataset [20] as well as the Microsoft Scalable Noisy Speech Dataset [15]. In total the volume of the sound data we uploaded in the Edge Impulse platform for the purpose of model training and validation added up to 8 hours and 23 minutes of audio, while each sound sample was a few seconds long.

The customization of the machine learning model we used for bird audio recognition in the Smart Birdhouse was done through a series of blocks/layers in the Edge Impulse platform such as setting a window size of each sound sample that will be used in training. In our case the window size we used was 3500 ms since we believe the it is an appropriate time window for the sound recognition of species that have long acoustic spaces between their singing or calls to other birds, as is the case for the Great Tit. We also chose to use an MFCC (Mel Frequency Cepstral Coefficients) block which is responsible for the extraction of useful characteristics from the training data and can be used for the task of bird sound classification [13]. For the classification task our model used a neural network that included three one dimensional convolution layers with three neurons each. In Fig. 3 we present the detailed architecture of the neural network used as was visible in the Edge Impulse platform.

A linear diagram showing the layers of the neural network for audio classification. The network consists of a flat input layer of 2275 features, a reshaping layer of 13 columns, a one-dimensional convolutional layer with 8 neurons and kernel size of 3, a 0.25 dropout layer, a one-dimensional convolutional layer with 16 neurons and kernel size of 3, a 0.25 dropout layer, a one-dimensional convolutional layer with 8 neurons and kernel size of 3, a flattening layer and finally an output layer of four classes.

The above neural network was trained with the data samples we described above for 50 epochs with a learning rate of 0.005 and a validation set size of 20 percent. The training accuracy of the above described machine learning model was 88.4 percent while the testing accuracy was 85.40 percent which indicated acceptably good performance of our sound classifier for unseen data, although the test data volume has been limited.

4.3 Data handling by the dedicated server

The collected atmospheric values and the photos/sound data that were collected from the smart birdhouse are sent to a dedicated server through the use of the MQTT communication protocol as we discussed in previous section. The data are specifically sent to an MQTT broker that runs on that dedicated server and each type of data has a different MQTT topic where its being sent to. Following that the Node-Red platform, which also runs on the dedicated server, will convert the audio and image data to base64 format so they can be saved later to the ThingsBoard platform where they will be sent along with atmospheric data collected. Also all data are sent to the ThingsBoard platform through http request nodes by Node-Red for storing as well as to provide a relevant dashboard hosted there with statistical info. In more detail, atmospheric data like temperature, humidity, ambient light level as well as a count of how many times the PIR sensor detected movement inside the smart birdhouse are being used by a relevant ThingsBoard dashboard to produce historical statistics about them or provide live information about these data like shown in Fig. 4.

Two screenshots from the Thingsboard platform interface, showing current temperature, humidity and lux values, and line charts of historical data for the same parameters.

4.4 The Digital Twin of the Smart Birdhouse

The Digital Twin system is designed to play back photos and sound data that were collected from the Smart Birdhouse in near real time after they have been processed and collected by the dedicated server (Fig. 5). The Digital Twin is comprised of a Raspberry pi microcomputer which will be equipped with an LCD screen for viewing photos as well as reproducing sound data that were collected in appropriate times from the birdhouse. This playback is achieved through the use of a Node-Red flow that uses appropriate Node-Red nodes to achieve playback of audio and photo data.

A bare-bones construction of the digital twin, consisting of an LCD screen mounted on top of a Raspberry-Pi microcontroller. The Raspberry's audio output is connected to a pair of headphones.

The playback of photos and sound in the Digital Twin is being achieved by the use of a bash script that uses a image viewer for linux named feh viewer[5], while at the same time through the use of the mpg321[6] command-line audio player a collected audio is played back at near real time in relation to when it was captured. In this way the Digital Twin can reproduce the events inside the smart birdhouse in photographic and audio form by pushing these to learners when detected, while the ThingsBoard platform we presented in the above section can show the atmospheric data collected as well as statistics about them, requiring explicit pull of the data from the learners.

For the evaluation of the smart birdhouse we deployed it in a controlled outdoor environment for a period of two weeks during the month of August in 2022 where it run for approximately from 8 o clock at morning till 9 o clock at night while it was connected to a nearby power supply which powered the raspberry pi device as well as the other sensor devices powered by it. While the smart birdhouse was running, it executed the Python script that allowed for data collection from its inside as we mentioned in above sections. We observed that the smart birdhouse run during that period with no problems as well as it was providing all the data it was supposed to without significant delays, and it was immediately available for viewing in the ThingsBoard platform or the Digital Twin in the case of atmospheric data or sound/photo data respectively. During the testing period of the smart birdhouse no bird visitors entered so we tested the photo and sound capturing capabilities of our smart birdhouse through the use of a mobile phone that was playing YouTube videos for one of the bird species that can enter the birdhouse ("housesparrow","greattit","greenfinch"). With the use of these videos and by us triggering manually the PIR sensor with our hand we verified that our audio classifier was correctly classifying bird audio to its correspondent class while there was not such audio it was classified as noise. It is worthwhile to mention that during the two week testing period the smart birdhouse managed to collect 96KB of atmospheric data which includes temperature, humidity and ambient light values. From these observations we believe that the Smart Birdhouse system can work with reliability and trustworthiness for a long amount of time in the outdoors while collecting and transmitting relevant environmental data.

In this section we would like to summarize our presentation of the novel smart birdhouse system we propose as well as include our own conclusions from its development and testing. Drawing inspiration from the current developments of the 4^th industrial revolution, we used the idea of one of its core driving technologies (Digital Twin) to develop a novel smart birdhouse system for the purpose of environmental education. The novel smart birdhouse system we proposed consisted of three main parts, a smart outdoor birdhouse device that will collect atmospheric data as temperature and humidity from its interior as well as photos/sound when the presence of a bird is detected, a dedicated server that will receive this data from the smart birdhouse and serve historical data dashboards to users, and finally a realtime Digital Twin device that will play back collected photo/audio data. We evaluated the above system by deploying the smart birdhouse outdoors in a controlled environment and verifying its correct functional operation. From the above we conclude that the novel smart birdhouse system we propose is a reliable and trustworthy prototype that can be used for the purposes of environmental education, especially through the use of Digital Twin devices for raising environmental awareness of young children through the playback of nature-related images and sounds.

Our prototype can be made energy autonomous through the use of solar panels as well as batteries that can be used to power these panels which will make the smart birdhouse deployable in remote natural settings. A further advantage is that one outdoor unit can feed data to multiple indoor units, therefore presenting engagement opportunities to multiple classrooms and learners. Of course, any indoor unit can be connected to any outdoor unit, allowing transitions between various sites of interest, and exchanges between classrooms. Following our prototype proof-of-concept, we plan further research with learners and educators for finding the best operating parameters of this novel birdhouse system, so it can be incorporated in school environments.