TempMesh – A Flexible Wireless Sensor Network for Monitoring River Temperatures

There exist a few sensing technologies capable of the type of data collection required for the project. Forward-looking infrared (FLIR) cameras can be used to image temperatures at the necessary spatial density, either aerially or from channel margins. Unfortunately, achieving the required spatial extent was infeasible, as it would require that we install dozens of FLIR cameras or move them constantly. They also can only image the water’s surface [9]. FLIR deployments also fail to meet the temporal extent needed for this study, as they would require considerable maintenance and power over the course of the six-month juvenile residency period.

Fiber-optic networks can sample large distances (~13 km) at a time, with about 1 meter spatial resolution [10]. As a result, fiber optic temperature sensing met the necessary spatial density and extent for our work, with the fiber optics repeatedly crossing the river in a zig-zag pattern. Unfortunately, fiber-optic temperature sensing was not feasible for this study due to the required lasers, which use considerable amounts of power and the need for continuous, on-going upkeep and maintenance. This was impractical given the access issues of the study sites and the sampling duration.

It became clear that the best option for temperature monitoring was the use of electronic temperature sensors. We initially considered data-logging sensors, but moved away from these because the lower Yuba River experiences extreme flood events that can transport trees, large boulders, and other detritus down-river. These events can damage, bury, or dislodge temperature sensors. Since there is no way to remotely recover data from low-cost in situ logging sensors, destructive events eliminate all data since the previous download, and data from flood events is lost.

Ultimately, we opted to construct a wireless sensor network that sensed data using off-the-shelf temperature sensors. We selected the Campbell Scientific CS225-L SDI-12 temperature sensor cables, which were environmentally sealed and protected all sensor hardware by housing it in a thick, sealed cable assembly. These assemblies included individually addressable, digital temperature sensors, allowing for long cable runs without concern about voltage loss or other drift that could alter sensor readings. These sensors were sufficiently accurate for the study (\(\pm 0.2^{o}C\)). Sensor assemblies were designed to contain five sensors, evenly spaced either 5 m or 10 m apart along the length of the cable, depending upon the river width at a particular location. Sensor cables also had between 10 m and 100 m of additional cabling on the end to allow for transmitters to be concealed in riparian foliage. These sensors were tethered to anchors buried in the riverbed. Sensors were initially designed to withstand large changes in the flow of the river, with sacrificial anchors, which would detach under destructive flows and allow the sensor cables to survive. Despite these precautions, it proved nearly impossible to design the sensor systems to survive impacts from massive detritus such as logs carried down by the river during the highest of flows.

The Campbell Scientific CS225-L temperature sensors were integrated with a wireless mesh network built using inexpensive, low-power devices. Each device was equipped with a radio with reasonable range to aggregate the temperature data from the sensor strings to one aggregation point from which the data could be back-hauled to a server on the University of California, Davis, campus using a cellular connection.

There are four main technologies for implementing wireless mesh networks [22], based on (1) IEEE 802.11s, (2) Bluetooth Low Energy (BLE), (3) IEEE 802.15.4, and (4) LoRa (and the corresponding MAC layer protocol, LoRaWAN). The key advantage of devices built using these technologies is that they come with protocols for node discovery, topology formation, and data forwarding and routing. These technologies also have high bandwidth. The range for BLE-based devices is about 30 m. While BLE’s relatively high bandwidth at this range is optimal for its intended application, it was infeasible for the present study, as 30 m node ranging was inadequate and would have required far more mesh nodes than could be installed in the given terrain. IEEE 802.11s [23], IEEE 802.15.4 [24], and LoRa [25] are designed for Internet of Things (IOT) applications in industrial setting or Smart City applications where either power is readily available or batteries can be easily recharged or replaced. These technologies were not designed to minimize power use. We wanted to use 433 MHz band because of its superior propagation characteristics outdoors compared to 2.4 GHz or 900 MHz, despite the negative effects of having a larger Fresnel zone. As a result, the 433 MHz band has gained momentum for machine-to-machine communications using low-power wireless technologies [26]. DASH7 and IEEE 802.15.4f are two standards that have been developed for the 433 MHz band. At the time of the deployment, LoRa had not yet added support for the 433 MHz band. Our application required reliable data transmission in an energy-deficient environment. We prioritized power savings and data fidelity over bandwidth, latency or other considerations, because the raw temperature data occupied only a few bytes, which meant each sensor node could generate small packets every 15 minutes.

We opted to use wireless communication operating at 433 MHz, because the frequency struck a reasonable balance between range and penetration. It also was an unregulated frequency band—requiring only a Federal Communications Commission (FCC) experimental license. Using low-power transmitters like ours, operating at 433 MHz, 500 m range was achievable with simple, low-cost whip antennae. At this range, 433 MHz transmissions can penetrate vegetation while reflecting off of solid surfaces. In a river, the steep embankments leading up to the floodplains and levees can create something of an echo chamber, which we found to enhance transmission ranges.

The nodes of the wireless mesh network were implemented using WizziMotes [27], an IoT device built on top of the Texas Instruments (TI) CC4305137, an MSP430-based micro-controller unit (MCU) with integrated 433 MHz wireless networking. The MSP430 family of MCUs are ideal for this application, as they are low-power and can be switched to ultra-low power modes, which significantly extend battery life [28, 29]. We used WizziMote because they were among the only devices available at the time that operated in the 433 MHz band and implemented some aspects of the DASH7 protocol [27, 30].

As a System-on-Chip (SOC), MSP430 provides integrated peripherals for a variety of battery-operated wireless applications. The operating modes take into account three different needs: ultra-low power, speed and data throughput, and minimization of individual peripheral power consumption [28, 29]. Built with Low Power Modes (LPMs; LPM0: \(80 \mu A\), LPM2: \(6.5 \mu A\), LPM3: \(2 \mu A\), LPM4: \(1 \mu A\)), MSP430 can preserve energy by shutting down respective clocks on the processing chip and later returning to Active Mode (AM: \(160 \mu A\), radio in RX: 15mA, radio in TX: ~30mA depending on transmit power) through enabled interrupts in less than \(6 \mu s\) [28, 29]. The state of execution is saved on the stack and is restored unless altered during the interrupt service routine. State and memory are maintained and, as a result, the network is able to store data while nodes are put to sleep. This was an important requirement given that we had intermittent link failures.

The network protocol stack of WizziMote is built around the DASH7 standard [30]. The data communication is simplex, i.e., the network interface can only operate as a transmitter or as a receiver at one time. The radio channel bandwidth is 1.74 MHz, spanning from 433.05 to 434.79 MHz. While DASH7 offers different classes of communication channels, in this project, we used the normal category [30]. It provides eight channels, each with a data-rate of 55 Kbps. In our experimental analysis with the WizziMote, the neighboring channel interference was high. As a result, we used only four non-overlapping channels.

The network was deployed in November 2018 on the 3 km study reach of the lower Yuba River. The sensor network remained functional through May 2019. The study reach was immediately downstream from Daguerre Point Dam (Figure 1: not featured, just upstream of the top-right of the map) near Marysville, in California’s Central Valley. The river is a multi-threaded river. The primary channel of the river is approximately 100 m wide in the study reach, with a bankful flow of roughly 141.6 \(m^3/s\). The river has a course riverbed of large cobble, allowing for considerable hyporheic flows through the gravel. The river is heavily modified due to extensive aggregate gold mining in the region. This reach of river was in fact re-located a century ago, moved around the gold-fields and to the dam. As a consequence, the river is quite treacherous—surrounded by large gravel mounds. Further, the movement of the river was relatively recent, so there has been little ecological succession that would yield sedimentary organization. Indeed, much of the floodplain contains large, exposed cobble. It is a very difficult area to navigate, and low sediment organization, substantial hyporheic flows, high velocities, large cobble size, and forced channeling all contribute to a particularly dynamic study reach, prone to re-organization and movement during flood events (Figure 3). As such, sensors were placed within the river in places that were accessible by foot, without the need of a boat. Sensors were placed, orthogonal to the direction of flow, with effort made to run them through as many different geomorphic features as possible. Because of the limitations of this dynamic system, sensor placement was not random (though random number generators were used to select approximate location), field crew safety required adaptation to place sensors in areas that could be reached without workers being harmed or carried away by substantial flows.

There were three types of packets: data (D; presently temperature data), health (H), and test (T). Temperature packets consisted of a node ID (the unique identifier of the sensor node from which the data originated), sequence number (a counter that incremented at the sensor node for each transmitted packet so duplicate packets could be identified), sensor type (an ASCII character to characterize the sensor—presently “T” for temperature), message type “D” for data, first sensor ID, last sensor ID (sensor nodes were designed with multiple sensors per node—presently five—using the first and last sensor ID, we could infer whether sensor readings were ascending or descending in their ordering and allowed for the pinpointing of malfunctioning sensors), up to 20 bytes of data (in this application, up to 10 temperature readings), and two bytes of 0xFF to signify end of message.

A health packet (H) contained only metadata: node ID, sequence number, sensor type of “R” for recovery (sent when a node restarted or regained power after a power loss), and message type of “H” for health. Sent at one-hour intervals, the reception of health packets ensured that the network was functional between the originating nodes and the gateway. These packets were particularly useful during periods when network links failed. All nodes (including relay nodes) generated health packets, which allowed us to localize failures to a specific node or region of the network. A testing packet (T) was used for debugging purposes. Testing packets had a structure of node ID, sequence number, sensor type “T,” and data type “T”; they were generated by handheld nodes, carried by researchers to test wireless communications between nodes as they were being deployed in the field, ensuring optimal node positioning and placement.

The maximum packet size was 28 bytes, set statically by the sensor node when creating packets. The first four of these 28 bytes were reserved for the packet metadata. The actual packet could be smaller, depending upon the amount of data collected by the sensors connected to the sensor node.

Due to the limited access to the lower Yuba River, the lack of access to power, and the threat of vandalism posed to visible structures like solar panels, it was imperative to minimize power use. The sensor nodes aggregated temperature data from the sensors and transmitted to the next upstream relay node every 15 min. This implied that all sensor nodes could be powered down to low-power states with a duty cycle of 15 min. A related issue was that nodes that transferred data to the same upstream node use the same frequency. This led to collisions, which needed to be prevented to minimize power use.

Energy-efficient MAC protocols have been extensively studied in the context of wireless sensor networks with a sleep-awake duty cycle. Broadly, they can be categorized into four classes: synchronous, asynchronous, hybrid, and random [31]. Due to the lack of real-time synchronized clocks in the nodes, synchronous approaches such as T-MAC [32] and hybrid approaches that combine synchronous and asynchronous approaches could not be used. It was further impractical to implement any wireless time correction like GPS, as GPS reception can be impacted by vegetation, and this interference can yield errors in both positional and timing signals [33]. Additionally, GPS-based timers used considerably more power than was available in our installation. GPS receivers were also more expensive than our budget allowed. With low duty cycle sensors (such as those implemented in this project), it has been found that idle listening can be reduced by having the transmitting node be the active entity when synchronizing with the receiving node [31]. A common way for this to be achieved is for the sender to—when it has data—transmit a preaamble that lasts for a length of time that is long enough to overlap with the short duration of time for which the receiving node wakes to listen for the preamble. This overlap is guaranteed even if the clocks in the transmitting and receiving nodes are not synchronized. If the receiving node detects the preamble, then it stays awake to receive the data. This is the approach adopted in B-MAC with additional optimization in X-MAC [31]. Recently, a randomized algorithm based on the Birthday protocol [34] has also been implemented [35].

The MAC protocol adopted by WizziMote was similar to B-MAC [36]. In the WizziMote, time was measured in “ticks,” with each tick equal to \(1/1,024\) seconds. For our deployment, we set the sleep duration to 900 ticks, while any advertisements were transmitted for 1,000 ticks. This ensured that receiving nodes woke up before the end of the advertisement period of the transmitting node. During the 1,000 ticks, the transmitting node continuously sent packets containing the remaining time of advertisement (in ticks). The receiving node used this information to synchronize with the start of the data transmission. At the transmitting node, the packet was transmitted immediately after the completion of the advertisement. Once the data were transmitted, the transmitting node switched the radio interface to the receive mode to receive the acknowledgment. At the receiving node, if the data passed the checksum, then the node immediately acknowledged the transmission on the same (backward) channel. Notably, the acknowledgments did not use advertisements. The data transmission and the acknowledgment were performed in one single operation [37]. Furthermore, nodes transmitted data regardless of whether or not any node received the advertisement due to a link failure. If the transmitting node did not receive an acknowledgment, then it reattempted the transmission in between sleeping until the receiving node acknowledged the data packet. The reattempts were guided by an exponential back-off algorithm (Section 4.5).

The energy savings of this approach was derived from the fact that the sender advertised only when it had data to send and the receiver was only awake for only a small fraction of the time. The use of advertisement to synchronize nodes significantly increased the chance of collision at locations where multiple branches of the network converged (i.e., where multiple nodes transmit to a single node). At these locations, multiple nodes could be advertising, transmitting, or acknowledging at the same time. This yielded considerable interference in laboratory experiments. These transmission collisions substantially increased transmission times and diminished network fidelity, filling the distributed network storage (discussed in Section 4.5). To avoid these collisions, while still preserving the advertisement-based synchronization, we implemented cross-listening: The transmitting node listened on the channel before it started to advertise on the channel. This was similar to Carrier Sense Multiple Access (CSMA). The main difference was that traditional CSMA with collision detect (CD) and CSMA with collision avoidance (CA) required the radio interface to operate in full-duplex mode, i.e., able to transmit and receive at the same time; whereas the WizziMote radio interface operated in simplex mode. Thus, in cross-listen, the transmitting node only listened for the advertisement and not a general carrier signal before any data transmission. By minimizing channel conflicts, reducing contention, and decreasing transmission attempts, cross-listening also reduced duplicate packet transmissions, decreased the time for a packet to reach the gateway node, and reduced network-wide power use.

Routing protocols in the context of wireless sensor networks have been extensively studied [38] and there is a standard routing protocol defined for low-power, lossy networks like wireless sensor networks [39]. There were multiple constraints that limited the design and implementation of the routing protocol. First, WizziMotes did not have any in-built node discovery and routing algorithm. As a result, any routing algorithm had to be implemented from scratch. Besides the complexity of many of the standard algorithms, there was limited memory in each WizziMote (see Section 3.3). Additionally, there were both topographical constraints (dense foliage and sharp embankments at many locations) and constraints on the number of potential locations where relay nodes could be deployed. Consequently, the network of the relay nodes was a sparse network and there were only a few routes between the sensor nodes and the gateway node. Near many of the sensor nodes there was only a single reachable relay node.

There were a number of conflicting constraints imposed upon our network; in balancing them, we opted for simple static routing and implemented network storage to account for temporary link failures. The locations of the temperature sensors were fixed by the scientific requirements. Specifically, we needed to measure temperature along a 3 km river reach at regularly spaced locations. At each location, we needed multiple, evenly spaced (5–10 m) sensors. The sensor nodes that aggregated the cross-channel temperature at different locations along the river were constrained by the river’s inherent geometry. The harsh terrain prevented a very dense deployments of relay nodes and was further complicated by the incidence of fouling and vandalism. The equipment and deployment budget limited the number of relay nodes. We therefore deployed a relatively sparse network of relay nodes. Furthermore, the 433 MHz radio that was implemented in the WizziMotes could reliably discriminate only four out of the seven available channels. For all of these reasons, we implemented a simple static routing strategy and used network storage to mitigate link failures.

To simplify the routing of packets and reduce energy use by eliminating the need for node discovery and distributed topology creation, all routes in the network were set statically. To minimize interference, nodes that were in the hearing range of each other but on different routes were assigned different channels. The channels were reused in links that were far apart. Developing the network with static routing simplified the implementation and provided greater control over the exact traversal path of packets. This came at the cost that there were no alternate paths for packets when field conditions changed. But the limited deployment meant that in most places on the river, no alternative path existed.

A potential approach to deal with link failures would be to set up static back-up paths. If the primary path broke, then the back-up path could take effect to maintain the connectivity of the network. However, this option would only be viable with a dense deployment of relay nodes. In fact, for a dense deployment of relay nodes and with more power, processing, and memory in the relay nodes, a dynamic routing protocol would be beneficial. The overall topology of the sensor and the relay nodes had an unbalanced tree structure with the gateway at the root node (Figure 1). It was only at merge points that nodes in tree branches were close enough to hear each other. We used cross-listen at these points to reduce collisions.

By doing a pilot study with a small, experimental field deployment, we found that weather, distance, local topography, vegetation, antenna position and direction, and other factors reduced link fidelity and intermittently hindered transmissions. Further, transmission were lost to collisions at points where network branches merged. For times of diminished network stability, we designed modules that minimized repeated re-transmissions while maximizing the throughput of unique packets. This was achieved by exponentially backing off of the transmitter when transmissions failed. For example, if the base transmission rate was every 4 s, then after subsequent unsuccessful transmissions, the base was binary exponentially increased to 8 s, 16 s, 32 s, and so on. When the link quality was impacted by weather, it often remained so for some time. Thus, the exponential back-off (as opposed to a linear back-off) reduced the number of unsuccessful transmission attempts, and saved power. While the node slowed down its transmissions, it still received packets until its queue was filled. This, in turn stored as many packets as possible within the relay nodes in the network. Once the link became reliable—or in other words, once the node successfully transmitted a packet again—it quickly recovered from exponential back-off by immediately resetting its base to four seconds.

We have separated the network storage algorithm into two parts (Algorithm 1): The first was done by the transmit thread, wherein it retrieved the number of unsuccessful packet transmissions, and then the receive thread updated the back-off timer. This back-off timer reduced the frequency with which transmissions were attempted. In our current implementation, the delay began at four seconds between transmission attempts and doubled for each transmission failure, up to a maximum of 30 minutes. This maximum ensured that each node attempted a transmission at least every half-hour. Any successful transmissions restored the delay to four seconds.

These transmission delays coupled with the 10-slot queues in each relay node coalesced to yield up to a few hours of distributed network storage (depending upon the design and topology of the network). This data retention was crucial for scientific sensing, which could generate a steady stream of sensor-derived data packets. Their timestamps were reverse-calculated once they successfully reached the gateway (Section 6).

Nodes in the network utilized the WizziMote’s Kernal Abstraction Layer (KAL), which maintained the illusion of multi-threading at a high level by encapsulating the Contiki proto-thread library [27], an open-source operating system that offers dynamic loading of application code onto embedded systems. While similar to a threaded architecture, pseudo-threads cannot run in parallel. Proto-threads are lightweight and stack-less, providing a blocking context on top of an event-driven kernel. Sequential flow of control is implemented with macros that save the processing states of functions without using complex state machines or full multi-threading. Proto-threads were particularly well suited for memory constrained devices (WizziMotes had only 4 KB of memory). Another useful feature is that preemptive multi-threading is implemented in the proto-threads as an application library. Consequently, multi-threading can be optionally linked with programs that explicitly require it [40].

Two types of events—timer and radio—controlled a pseudo-thread’s execution flow. A pseudo-thread was always in one of the following states: inactive, processing an event, or waiting for an event. The WizziMote radio library was written with only one radio buffer. While both the transmit and receive threads could be active at the same time, they could not process radio events simultaneously because of the shared buffer. To prevent conflicts over radio resources, these threads were scheduled sequentially (transmit after receive).

A finite packet buffer, implemented as a queue, connected the threads and acted as shared storage. Packets received from the receive thread were inserted into the queue, which were then removed from front of the queue by the transmit thread. To reduce network load from re-transmissions that resulted from unheard acknowledgments, the receive thread compared each received packet with the packet received before it. Identical, sequential packets were not inserted into the queue.

In each relay node, there were three threads: the management thread, the transmit (TX) thread, and the receive (RX) thread (Figure 4). At points of intersection in the topology, we also introduced the cross-listen thread for collision control.

For nodes to survive in the field with limited battery power, they were put to sleep when not transmitting or receiving. During this sleep, each relay node was pulled to Low Power Mode 3 (LPM3), which disabled the CPU (MCLK) and the high frequency clock (SMCLK), leaving only the 32 kHz low-frequency crystal clock ACLK powered [29]. In the main() of each relay node, a timer of 900 ticks was initialized and started, which created an interrupt and returned the node to Active Mode.

A few lines later, after closing the radio layer and the serial, the node was put into LPM3 and global interrupts were enabled. The node woke up when the main timer timed out.

This process was repeated after every cycle of receive and transmit.

To determine the power used by our relay nodes, we conducted an experiment that logged the current drawn by relay nodes. We did this by setting up three nodes: a dummy-sensor node that generated numerous packets at regular intervals, a relay that transmitted the data, and a gateway node that received the data from the relay. We set up the dummy-sensor node to transmit at regular intervals of 4, 8, 16, or 32 seconds. The relay node was connected to a 3.600 VDC source, through a high-speed, logging bench-top multimeter to measure draw. We measured voltages both above and below this logging meter to ensure that burden voltage was not too large and that supply voltage to the relay node stayed above 3.500 VDC. These voltage monitoring multimeters were both put into their high-speed max/min modes, which measured at least once every 1 ms. These multimeters were recently calibrated; one was an Amprobe AM-270 and the other a Fluke 87. The power source was a recently calibrated Power Designs 4010. The current was logged by a recently calibrated Rigol DM3051. This meter was capable of 500 samples per second, with storage for 200,000 samples—yielding 6 minutes and 40 seconds of continuous sampling.

Each of the relay node’s states were reflected consistently in the data (Figures 7 and 8). Low-power sleep states used 2–3 \(\mu\)A, active mode used about 5 mA, and receive and transmit modes used 20–30 mA (Figures 7 and 8). These values were consistent with the anticipated power use from the CC430F5137 datasheet (Section 3.3) [28, 29].

To test the effectiveness of the network storage, we created a simple experimental network in the laboratory that simulated link instability and observed the network recovery and loss. We considered a linear network of a sensor node connected to a gateway node through two relay nodes. The sensor node transmitted data every 30 seconds, and the relay nodes had a packet queue of 10 packets. The link between the second relay and the gateway was shut down with increasing duration of 10, 100, 600, and 1,000 seconds. Each link failure was followed by 60 minutes of potential recovery time. Recall that the definition of network recovery in our network storage algorithm is equivalent to a successful transmission through the broken link (Algorithm 1). Hence, we defined the time for network recovery time as the time difference between the link failure and the receiving of the first packet after the link became available again. Note that there existed an idle time period during which the network has physically recovered, but was undiscovered until a packet traversed the previously broken link. For the purpose of this application, we considered the link to be dysfunctional during this idle time period, because the network health was dependent on the overall traversal of packets in the network rather than the individual link health.

Figure 9 shows the box plot of the recovery time for different durations of link failure. As the link breakage time increased, the time to recovery also increased. As the link was broken for longer periods of time, the delays in between transmissions increased exponentially and so did the recovery time of each node. The queue size also influenced the back-off rate of the network. When the queues of connected relays filled, the overall recovery time of the network increased as multiple relay nodes’ back-off durations compounded.

The rate of packet creation was significantly greater than in a real-world deployment, which ensured that the packet queues would fill while the link was broken (particularly during the longer duration link failures). The purpose was to test the back-propagation of packet storage. This effect was observed by looking at the losses. For any given queue size, breaking any link in the network had no effect on the packet loss if the down time was relatively low. Once the queues of all available relays were filled, the packet loss increased in proportion to the link down time. In a real-world implementation, the packet frequency would be much smaller, and the number of relays much larger, granting the network more storage over a longer duration of time, minimizing the probability of packet loss.

The distributed network storage implemented in this network resulted in the accumulation of unknown delays during the transit of sensed temperature data. The network contained no real-time clocks, and packets were timestamped upon receipt at the gateway, after any applicable network delays. It was necessary to re-align these packets with real-time. Re-alignment was feasible, as each packet contained an 8-bit packet ID, which was sequential (rolling over at 255). We performed the re-alignment in a two-step process, first identifying improperly timestamped packets (Figure 10(a)), then re-aligning them (Figure 10(b)). The central idea underlying both the identification and re-alignment of packets was the use of neighboring packets to estimate the timestamp for each packet.

While packets from other sensors may be useful for aligning packets in situations where transmission delays propagate from sensor or network drop-outs, most of our network failures were the result of power-failures at the gateway. Since timestamps were generated at the gateway, and most link failures propagated from the gateway (as it was the only solar-powered unit in the network and was most prone to power loss during inclement weather), we did not use packet timestamps from other sensors to align packets, opting to instead use packets from each sensor to exclusively align timestamps from that same sensor.

Each sensor generated a data packet about every 15 minutes, and each packet contained a sequential packet ID. It was therefore possible to estimate the timestamp of any individual packet (P) using the timestamps of neighboring packets (X of length N). By taking the difference in packet ID and multiplying by time between sensor readings (i.e., 15 min), we were able to estimate which packets were properly aligned. The method was robust to many different modes of packet delay, because it treated most aspects of packet timings as variable. To this end, we knew that the time between sensor readings (T) was determined by a crystal, and never synced to any clock, and therefore fluctuated with temperature. To improve timestamp estimates, we began by setting T=14 min, then increased it in five-second increments until the model heuristics (Figure 10) indicated that we had reached an optimal timing for each packet. When evaluating whether timestamps were aligned (Figure 10(a)), we evaluated the range of estimates, the distance of the estimate from the observed packet timestamp, and the number of estimates used to generate the estimated timestamp. In each step, we also removed the minimum and maximum estimate in an effort to improve the mean estimate and decrease the spread of estimates. If the estimates derived from a set of neighboring packets failed to meet the appropriate model heuristics, or if after filtering out minimum and maximum, the number of estimates fell below L (L=6, presently), then we increased the number of packets considered (N) to include additional packets before and after the packet in question. In this way, we prioritized using the packets closest to the packet in question, expanding the window only when estimates were not sufficiently robust using a smaller numbers of packets. If N reached 72 (36 packets before and after P), then we marked packet P as bad, meaning that we could not create a sufficiently robust estimate of the correct timestamp for P. Once each packet P had been evaluated, any marked bad were removed from the data and the algorithm started over, evaluating all remaining packets not identified as bad in previous rounds and using only the remaining packets not already marked bad to evaluate each packet within the reduced dataset. This repeated until the list of bad timestamps grew by less than 5% in a given round. At this point, we considered the list of bad packets to have been identified for that sensor. These bad timestamps were then fed into a timestamp re-alignment algorithm (Figure 10(b)).

The process for timestamp re-alignment was similar to that of bad timestamp identification. We used neighboring packets that were not identified as bad (Figure 10(b)). Instead of using the N-nearest packets as above, we instead used good packets arriving within H hours of P. H was initialized at 7 hours before and after the documented timestamp of P, increasing iteratively until the number (N) of packets (X) used to estimate the timestamp of packet P was at least 20, with H having a maximum of 48 hours. Once H was set and the N-packets (X) within H hours of P, the timestamp of P was estimated and the minimum and maximum estimates were eliminated until the range of estimates was less than 30 min. If this could not be reached, then as above, the inter-packet timing (T) was incremented by 5 seconds and the process was repeated. This was repeated until T reached 16 minutes.

The parameterization of variables and coefficients in our implementation were largely based upon the scientific questions at play. We were interested in hourly river temperature data. As such, we sought at least two measurements per hour. This is why we chose to set our rejection thresholds at 30 min for both for evaluating whether or not timestamps were correct, relative to timestamp estimates (i.e., mean (timestamp estimates) - observed timestamp) as well as ranges of timestamp estimates. Other applications of these methods should use parameters informed by relevant research questions.

We identified 9,594 incorrectly timestamped packets. Of these, 3,890, or about 40%, were able to be re-aligned using this procedure (Table 1). As a result of this identification and realignment, 40,012 packets were able to be used in further analyses of fluvial water temperatures. Notably, each packet contained five sensor readings, yielding a total of 200,060 properly aligned or re-aligned temperature readings from the sampling period (Table 1). For most sensors, timestamp re-alignment was most common for network delays of under 10 hours (Figure 11). Indeed, for every sensor except sensor E, 75% of re-aligned timestamps were shifted less than 10 hours (Figure 11). However, for sensor E, which was closest to the gateway, large delays were more readily fixed during timestamp re-alignment. As a result, the median re-alignment was much larger (over 10 hrs: Figure 11). Notably, timestamp re-alignment was feasible for up to 20 hours for at least some packets from each sensor (Figure 11). Timestamp alignment utilized the sequential packet ID, which rolled over at 255. Any timestamp alignment was infeasible after 63.75 hr, when the counter rolled over. Re-alignment also successfully corrected hourly mean temperatures, bringing them in line with what would be expected, with lowest temperatures in the early morning and warmest temperatures in the late afternoon (Figure 12). Prior to timestamp processing, the mean temperature was elevated in the early morning. Timestamp alignment rectified these high mean temperatures, as these incorrectly timestamped packets were shifted to the correct time.

We collected temperature data from November 2018 through May of 2019 (Figure 13). In this time, we collected 200,060 temperature measurements. Different sensors intermittently failed at different times, as a result of varying network failures (Figure 13). Many were damaged during large flow events (Figure 13: red lines). Notably, sensor E persisted longest, as it was nearest to the gateway and subjected to the fewest relay failures (Table 1). Upon re-aligning the data, we found a predictable temperature curve, with hourly mean temperatures peaking in the late afternoon and reaching a minimum in the early morning (Figure 12).

In total, 3,890 packets were re-aligned. We initially questioned the balance between the uncertainty implicitly introduced by timestamp re-alignment and the value of additional data. We came to the conclusion that re-alignment was crucial, because while it represented a relatively small number of packets (only about 11% of the total packets found to have been properly aligned: Figure 10(a)), timestamp mis-alignment was biased towards packets collected at night or during inclement weather, when network failures were most common. Eliminating 3,890 sensor readings explicitly collected at night and during rain would have hampered our ability to model river temperatures during these important times. This re-alignment was therefore crucial to the success of the network and the underlying ecological science (Figure 12).