[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
Channel-Hopping Sequence and Searching Algorithm for Rendezvous of Spectrum Sensing
Previous Article in Journal
Fault Detection and Diagnosis in Industry 4.0: A Review on Challenges and Opportunities
Previous Article in Special Issue
A Fusion Strategy for Vehicle Positioning at Intersections Utilizing UWB and Onboard Sensors
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 25
Issue 1
10.3390/s25010061
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Event-Based Visual/Inertial Odometry for UAV Indoor Navigation

by
Ahmed Elamin
1,2,*,
Ahmed El-Rabbany
1 and
Sunil Jacob
3
1
Civil Engineering Department, Faculty of Engineering and Architectural Science, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada
2
Civil Engineering Department, Faculty of Engineering, Zagazig University, Zagazig 10162, Egypt
3
SOTI Aerospace, SOTI Inc., Mississauga, ON L5N 8L9, Canada
*
Author to whom correspondence should be addressed.
Sensors 2025, 25(1), 61; https://doi.org/10.3390/s25010061
Submission received: 19 November 2024 / Revised: 19 December 2024 / Accepted: 24 December 2024 / Published: 25 December 2024
(This article belongs to the Special Issue Multi-sensor Integration for Navigation and Environmental Sensing)
Browse Figures
Figure 1
<p>Workflow of the proposed event-based VIO.</p> "> Figure 2
<p>DAVIS346 event camera.</p> "> Figure 3
<p>Study area camera calibration: (<b>a</b>) a 6 × 9 chessboard with a square size of 30 mm; and (<b>b</b>) an example of the detected pattern image.</p> "> Figure 4
<p>An office environment simulation layout.</p> "> Figure 5
<p>Simulated dataset: comparison of trajectories.</p> "> Figure 6
<p>Ground-based dataset: an example of feature detection and tracking on an events accumulated frame.</p> "> Figure 7
<p>Ground-based dataset: an example of feature detection and tracking on a standard frame.</p> "> Figure 8
<p>Ground-based dataset: an example of feature detection and tracking on combined events and standard frames.</p> "> Figure 9
<p>Ground-based dataset: comparison of trajectories.</p> "> Figure 10
<p>UAV used for the experiments. (1) DAVIS346 event camera. (2) NVIDIA Jetson Xavier computer. (3) Pixhawk 4 flight controller.</p> "> Figure 11
<p>UAV-based dataset: an example of feature detection and tracking on an events accumulated frame.</p> "> Figure 12
<p>UAV-based dataset: an example of feature detection and tracking on a standard frame.</p> "> Figure 13
<p>UAV-based dataset: an example of feature detection and tracking on combined events and standard frames.</p> "> Figure 14
<p>UAV-based dataset: comparison of trajectories.</p> ">
Versions Notes

Abstract

:
Indoor navigation is becoming increasingly essential for multiple applications. It is complex and challenging due to dynamic scenes, limited space, and, more importantly, the unavailability of global navigation satellite system (GNSS) signals. Recently, new sensors have emerged, namely event cameras, which show great potential for indoor navigation due to their high dynamic range and low latency. In this study, an event-based visual–inertial odometry approach is proposed, emphasizing adaptive event accumulation and selective keyframe updates to reduce computational overhead. The proposed approach fuses events, standard frames, and inertial measurements for precise indoor navigation. Features are detected and tracked on the standard images. The events are accumulated into frames and used to track the features between the standard frames. Subsequently, the IMU measurements and the feature tracks are fused to continuously estimate the sensor states. The proposed approach is evaluated using both simulated and real-world datasets. Compared with the state-of-the-art U-SLAM algorithm, our approach achieves a substantial reduction in the mean positional error and RMSE in simulated environments, showing up to 50% and 47% reductions along the x- and y-axes, respectively. The approach achieves 5–10 ms latency per event batch and 10–20 ms for frame updates, demonstrating real-time performance on resource-constrained platforms. These results underscore the potential of our approach as a robust solution for real-world UAV indoor navigation scenarios.

1. Introduction

Indoor navigation has recently received growing research interest due to its potential use in critical applications, such as medical care, search and rescue operations, and industrial management and inspections. In global navigation satellite system (GNSS)-denied environments, such as indoor environments, navigation is complex and challenging. Various sensors have been adopted for indoor navigation, such as inertial measurement units (IMUs), cameras, light detection and ranging (LiDAR), and ultra-wideband (UWB). The latter sensor is used to localize a system based on signal parameters received from fixed nodes of known locations. The most common signal parameters are the time of arrival, angle of arrival, and received-signal strength indication [1]. In [2,3,4], for example, a UWB-based system was utilized to navigate a UAV in indoor environments. However, UWB-based systems require additional infrastructure, which may not be cost-effective. LiDAR navigation, on the other hand, has traditionally been performed using either a 2D or 3D LiDAR sensor. Bachrach et al. [5] developed an unmanned aerial system (UAS) equipped with a 2D LiDAR sensor for navigation in unstructured indoor environments. Although 2D LiDAR navigation can operate in real-time, it cannot acquire height information and lacks the ability to construct a 3D map. On the other hand, in 3D LiDAR navigation, a complete real-time map can be constructed. An example of a commonly used 3D LiDAR navigation algorithm is LiDAR odometry and mapping (LOAM), which is ranked third on the KITTI data benchmark [6]. Light-LOAM introduced a computationally efficient LiDAR-based SLAM method designed for real-time performance on resource-constrained platforms [7]. In [8,9], a LiDAR/IMU fusion algorithm was proposed to localize a UAV in indoor environments. Despite LiDAR’s high accuracy and update frequency, the main challenges of using LiDAR sensors are the high price, large size, and power consumption [10].
Three types of cameras are widely used in visual sensor-based navigation: monocular, stereo and RGB-D. Monocular camera navigation is divided into three categories: the direct method, the feature point-matching method, and the semi-direct method [11]. Large-scale direct monocular simultaneous localization and mapping (LSD-SLAM), which estimates the camera motion using the gradient of the pixels, is one of the well-known direct methods [12]. Consequently, detecting feature points is unnecessary. Also, it can construct a semi-dense map. A typical representative of the point-matching method is the oriented fast and rotated brief SLAM (ORB-SLAM) [13], which was developed and formed on parallel tracking and mapping [14]. ORB-SLAM3 extends the original ORB-SLAM framework to support visual, visual–inertial, and multi-map SLAM for monocular, stereo, and RGB-D camera configurations, enhancing robustness and accuracy across diverse environments [15]. Semi-direct visual odometry (SVO) is an example of the semi-direct method [16], which combines direct and feature-point methods. Stereo camera systems are usually equipped with multiple lenses. Multiple lenses simultaneously capture the scene from multiple directions and then perform feature matching to restore more accurate 3D information about the environment [17]. The RGB-D cameras are equipped with a camera and an infrared transmitter and receiver, which can obtain the scene’s depth information [18]. At present, some mature RGB-D camera navigation solutions are available, such as RTAB-MAP [19] and dense visual odometry SLAM (DVO-SLAM) [20]. However, all these traditional cameras suffer from illumination changes in sensitivity, motion blur, and limited dynamic range.
A novel type of camera sensor called an event camera has recently attracted much attention from academia and industry [21]. It offers great advantages over conventional cameras, including high temporal resolution (in the range of μs), low latency, low power consumption, and a high dynamic range (140 dB vs. 60 dB) [21]. Event cameras are asynchronous sensors that represent a breakthrough in how visual information is acquired because they sample light based on the scene changes rather than at a rate an external clock specifies without any relation to the captured scene. Event cameras respond to brightness changes in the viewed scene asynchronously and independently for every pixel. Therefore, an event camera’s output is a variable data rate sequence of digital events. Each event represents a change in brightness at a specific pixel at a particular time.
In [22,23,24,25], it was shown that pose estimation is possible using only an event camera. For example, Kim et al. [22] proposed real-time 3D reconstruction and tracking, while Rebecq et al. [23] introduced a geometric approach for 6-DOF tracking. In [24], a method for range–visual–inertial odometry was presented, and in [25], an adaptive filtering approach was proposed to enhance the robustness and accuracy in dynamic environments. Events and inertial measurements were fused using a continuous-time framework [26]. However, this approach is not suitable for real-time applications due to the computational cost introduced by continuous optimization. Unlike traditional keyframe-based methods, this approach requires continuous spline fitting and optimization over an extended time window, leading to increased processing time and memory usage. This makes it unsuitable for real-time UAV navigation. In [27], an event-based visual–inertial odometry (EVIO) approach was proposed. An iterative expectation-maximization technique was used to track features in the event stream, and the feature tracks were then fed to an EKF filter to estimate the camera pose. EVIO can operate in real-time but for limited-speed motion. In [28], a real-time event-based VIO pipeline was proposed. This approach used keyframe-based nonlinear optimization to fuse the tracked features with the IMU measurements. They estimated the accumulated drift as the distance between the first and last positions, which reached 50 cm. In [29], event-based stereo visual odometry (ESVO) was developed, which addressed event-based stereo VO by maximizing spatio-temporal consistency across stereo event streams, allowing real-time 3D mapping and pose estimation even in high dynamic range and high-speed scenarios. Recent advances, such as ESVIO [30], leveraged event-based stereo VIO by integrating events’ depth, time–surface images, and IMU measurements in a sliding window framework, achieving high accuracy and robustness for state estimation in dynamic environments.
Feature detection and tracking are key steps in visual sensor-based navigation. Some commonly used image-based keypoint detectors, such as FAST [31] and the Harris corner detector [32], have been proposed for event cameras [33,34]. Also, histogram-of-oriented-gradients (HOG) detectors [35] have been developed for event camera recognition applications [36]. Generally, event-based trackers utilize binary feature templates, either predefined [37] or built from a set of events [38], to which they align events using iterative methods, such as the iterative closest point (ICP) [39]. However, those tracking approaches depend on motion direction and suffer from drift because of the event appearance variation over time [40]. Therefore, some approaches leveraged the advantages of frame-based and event-based combined cameras, such as DAVIS [41]. In [40,42,43], the algorithms automatically detect features on the frames and then track them asynchronously using events. Vidal et al. [44] proposed a state estimation pipeline that fuses data from an event camera, a standard camera, and an IMU. In their approach, the accumulated event frames are synced with the standard frames, where at each new standard frame a spatiotemporal window of events is selected. Subsequently, the features are extracted and tracked on both standard frames and event frames independently. Finally, the two sets of independent feature tracks are fused with IMU measurements through the back-end optimizer. They tested the system onboard a small UAV in an indoor environment under challenging conditions. The standard camera suffered severe motion blur at a relatively high speed, and the event camera showed reliable feature tracking. Therefore, the fusion of data from multiple sensors is essential for reliable and continuous localization for UAV indoor applications.
In this paper, an event-based visual–inertial odometry algorithm for event cameras is proposed. The proposed approach fuses data from an event camera, a standard camera, and an IMU for precise indoor navigation. Instead of tracking the features independently as in [38], features are detected on the standard frame images. Subsequently, to improve the tracking accuracy, the features are tracked on intermediate frames accumulated from events till the next standard frame. This ensures continuous tracking of the features in between standard frames, which enhances the VIO robustness. Finally, the feature tracks are fused with IMU measurements to estimate the sensor pose using nonlinear optimization. This paper is organized as follows: Section 2 presents the proposed event-based VIO approach. The experiments are explained in Section 3. Finally, conclusions are drawn in Section 4.

2. Event-Based VIO Approach

The overall methodology flowchart is shown in Figure 1, which summarizes the steps of the proposed approach. The proposed VIO approach is inspired by [28,44]. The proposed approach consists of two main threads, the front-end thread and the back-end thread. In the front-end thread, features are detected and tracked on the standard images. Between the standard frames, the events are clustered in spatiotemporal windows and accumulated into frames. These intermediate accumulated frames are used to track the features between the standard frames. To ensure robust tracking, 2-point RANSAC is applied to filter out outliers based on the relative orientation between the last keyframe and the current frame. Subsequently, the filtered feature tracks are triangulated to their 3D locations. The back-end thread fuses the IMU measurements and the feature tracks to estimate the sensor states continuously. This approach is implemented based on the open source library dv-mono-vio-sample, by iniVation [45].
In contrast to traditional SLAM pipelines that rely on continuous keyframe-based nonlinear optimization, the proposed approach focuses on lightweight optimization, processing events asynchronously to maintain smooth pose estimation between frame intervals. This design reduces computational overhead, making it ideal for low-power and embedded systems, such as UAVs operating in indoor environments. Features are detected on standard frames and tracked continuously on intermediate event-accumulated frames without requiring re-detection at every time step. By leveraging spatiotemporal windows of events and aligning them using IMU-based motion compensation, ensuring accurate tracking while minimizing drift. The fusion of these multi-sensor inputs enhances localization accuracy without the need for intensive optimization, providing a scalable solution for real-time navigation.
For every standard frame at time t k , the set of acquired events is clustered into overlapping spatiotemporal windows. The kth window, W k , which comprises a set of events, can be described as follows:
W k = e k S , … , e k S + N − 1 ,
where N is the window-size parameter and S is the step-size parameter, which determines the overlap between consecutive windows. Each spatiotemporal window’s duration is inversely related to the event rate. Each window, W k , is accumulated on an event frame I k by projecting each event onto the image plane as follows:
I k x = ∑ e j ∈ W k   δ x − x j ′ ,
where I is the intensity at pixel x , δ x is the Kronecker delta, and x j ′ is the event position after motion correction. Motion correction is important for reliable feature detection or tracking. Each event motion is corrected based on its timestamp by projecting the event, e j , to the reference camera frame as follows:
x i ′ = π 0 T t k , t i Z x i π 0 − 1 x i ,
where x i is the pixel position of an event e i ; π 0 .   is the projection model of the event camera acquired from intrinsic calibration; T t k , t i   is the transformation between the camera poses at times t k and t i , estimated using the IMU measurements; and Z x i   is the scene depth at pixel xi and time ti, which can be estimated using the current landmarks’ median depth.
The FAST corner detector [31] was used for feature detection along with a bucketing grid to ensure an even distribution of the features over the image. Whenever a new standard frame is captured or the feature tracks drop below a specific threshold, new features are detected. The FAST threshold was set to 10, which balances sensitivity to corner detection and robustness to noise. Lower thresholds increase the sensitivity but may detect more false corners, while higher thresholds miss important features. Empirical testing on both simulated and real-world datasets showed that a threshold of 10 provides an optimal trade-off between detection accuracy and computational efficiency. Then, the detected features are tracked across standard frames and event frames using Lukas–Kanade tracking (LKT) [46]. Thereafter, two-point RANSAC [47] is used to filter out the feature tracks’ outliers based on the relative orientation between the last keyframe and the current frame.
In the back-end thread, feature tracks from standard frames and the event stream are fused with the IMU measurements to estimate the sensor state over time. The back-end implementation is based on OKVIS [48]. In contrast to EKF-based filters, OKVIS is a full smoothing technique based on nonlinear optimization on selected keyframes. This approach outperformed state-of-the-art techniques in terms of accuracy [49]. For more details about the back-end pipeline, the reader is referred to [48].
The visual–inertial odometry problem is a joint optimization of a cost function, which can be formulated as
J = ∑ i = 0 1   ∑ k = 1 K   ∑ j ∈ J ( i , k )   e i , j , k T W r i , j , k e i , j , k + ∑ k = 1 K − 1   e s k T W s k e s k
This optimization function includes two weighted reprojection errors corresponding to the standard frames and the accumulated frames from the events in addition to inertial error terms, where i is the sensor index, k is the frame index, j is the landmark index, J ( i , k ) is the set of landmarks maintained in frame k by sensor i , e i , j , k is the reprojection error, W r i , j , k is the information matrix of the landmarks, e s k is the IMU error, and W s k is the information matrix of the IMU error. The information matrix can be described as follows:
e r i , j , k = z i , j , k − π i T C i S k T S W k l i , j ,
where z i , j , k is the image coordinates of landmark j on sensor i at frame k . The optimization is carried out on the standard frames as keyframes. Between frames, the IMU measurements along with the event frames that fall between frames are used to propagate the sensor state. Google Ceres optimizer [50] was applied to perform the optimization.
The current state is predicted based on the preceding state using the standard IMU kinematics and biases model presented in [51]. Then, the IMU error terms are calculated as the difference between the actual state and the predicted state. For orientation, a simple multiplicative minimal error is used.

3. Experimental Section

In this study, the DAVIS346 event camera [52] was used for data acquisition. The DAVIS346 integrates three sensors, namely an event camera, a standard RGB camera, and a six-axis IMU with a 1 kHz sampling rate. The event camera’s dynamic range is 120 db and a temporal resolution of 1 µs. On the other hand, the standard camera has a 55 db dynamic range. Both cameras have a 346 × 260 pixel image size, with a 113° horizontal field of view (FOV) and 97.7° vertical FOV. The DAVIS346 camera weighs 100 g without the lens, and its power consumption is less than 180 mA at 5V DC. Figure 2 shows an image of the DAVIS346 event camera.
The standard frame data were acquired at a 25 Hz frame rate, and the IMU data were acquired at a 200 Hz frame rate. Both cameras were calibrated prior to the data processing to determine the camera’s intrinsic parameters. The camera calibration was implemented using the DV software 1.3.1 [53]. A 30 mm2 6 × 9 chessboard was used in the calibration, as shown in Figure 3a. By moving the chessboard in front of the camera, the calibration module in the DV software detects the chessboard pattern. Figure 3b shows an example of a detected pattern image. After capturing enough detected pattern images, the software calculates the calibration parameters and the maximum reprojection error, which was approximately 0.08 pixels.

3.1. Simulated Dataset

To illustrate the effectiveness of the proposed approach, it was first evaluated on simulated data against the state-of-the-art algorithm ultimate SLAM (U-SLAM) [44]. An office environment was simulated using the Robot Operating System (ROS) and Gazebo, which spans an area of 600 m2, as shown in Figure 4. The simulation utilized the P230 UAV model, equipped with virtual sensors such as an RGB camera, an event camera, and an IMU. The DVS Gazebo plugin was employed to simulate the event camera data. Both the RGB and event cameras recorded images at a resolution of 240 × 180 pixels, with a horizontal FOV of 103° and a vertical FOV of 97.7°. The standard frame data were collected at 25 Hz, while the IMU data were recorded at a frequency of 50 Hz.
The proposed approach demonstrated significant performance improvements over U-SLAM in terms of position error metrics, as shown in Table 1 and Figure 5. It achieved smaller mean and root mean square error (RMSE) values across the x-, y- and z-axes compared to U-SLAM. Specifically, the mean positional error in the proposed approach was reduced by approximately 50% on the x-axis and 47% on the y-axis, while the RMSE exhibited comparable improvements, demonstrating a marked increase in overall accuracy. The maximum error recorded in the proposed method was also notably lower than U-SLAM, showcasing enhanced robustness and stability in simulated environments.
In addition to achieving significant reductions in the mean positional error and RMSE compared to U-SLAM, the proposed approach offers superior computational efficiency. By leveraging adaptive event-based tracking and selective keyframe updates, the system maintains continuous pose estimation with lower latency. Specifically, the proposed method achieved a processing latency of 5–10 ms per event batch and 10–20 ms for frame updates, compared to 15–30 ms per frame plus additional overhead for keyframe-based nonlinear optimization in U-SLAM. This reduction in computational demand makes the framework more suitable for resource-constrained devices, enabling deployment on platforms like lightweight UAVs. Furthermore, the efficient handling of events ensures robust performance in dynamic environments, providing consistent tracking between frames and mitigating the drift issues observed in systems that rely heavily on keyframes.

3.2. Ground-Based Dataset

In order to evaluate the proposed approach on real-world configurations, two datasets are presented: a ground dataset and a UAV-based dataset. Both datasets were acquired in an indoor environment following a predefined trajectory and ensuring a precise return to their initial starting positions. The ground-based dataset was captured through manual navigation with a DAVIS346 event camera. In the UAV-based dataset, a quadrotor platform equipped with the DAVIS346 camera was used for data acquisition.
The data were collected in an indoor environment with a complete absence of the GNSS signals. A predefined trajectory with precise coordinates was established and used as a reference. In addition, the trajectory was established to form a closed loop, which allows us to estimate the loop closer error and to use it as another evaluation metric. Three case studies were considered. The first of which was based on events only. In this case study, the tracking was achieved by accumulating frames from the event stream and then running the LKT frame-based tracker on them. Figure 6 shows an example of feature detection and tracking on an accumulated frame from the stream of events. The second case study, on the other hand, included the standard frames only. Figure 7 shows an example of feature detection and tracking on a standard frame. In both aforementioned cases, the feature tracks were fused with the IMU measurements using OKVIS in the back end. The third case study combined events, standard frames, and IMU measurements. In this case, the features were detected and tracked on the standard frame image, but to improve tracking quality, intermediate frames were accumulated from events and tracking was performed on those frames. Finally, the IMU measurements were fused with the feature tracks to estimate the camera states. Figure 8 shows an example of feature detection and tracking on the combined events and standard frames. The estimated trajectories from the three cases were compared to the ground truth.
Figure 9 compares the trajectories of the three case studies with respect to ground truth. Table 2 shows the error statistics of the position for the three navigation solutions. Table 3, on the other hand, shows the loop closing error for the three cases, which was calculated as the difference in coordinates between the first and last points. In Figure 9 and Table 2, it is noticeable that the event-only solution drifts significantly after a short time. The total RMSE of the event-only case was 2.33 m, while the loop closing error was 4.42 m. On the other hand, the standard frame-only solution sustained a relatively small error over a longer period, although it drifted by 2.45 m by the end of the trajectory. In addition, its overall RMSE was 1.45 m. Compared with the previous two case studies, the combined solution produced significantly fewer errors. The positioning RMSE decreased by 84% and 75% compared to the event-only and standard frame-only cases, respectively. The loop closing error of the combined case was 0.13 m for a 32 m trajectory, i.e., an accumulated drift of less than 1% compared to 13.8% and 8.0% in event-only and standard frame-only cases, respectively.

3.3. UAV-Based Dataset

To assess the developed approach in a real-world scenario, it was tested using a dataset that was captured by a UAV. The UAV was assembled using the DJI F330 frame and was equipped with a DAVIS346 event camera, a Pixhawk 4 flight controller [54], and an NVIDIA Jetson Xavier computer. Figure 10 illustrates how these components are rigidly fixed to the UAV platform, with the event camera positioned at the front, facing forward.
Similar to the ground-based dataset, the UAV flew indoors with no GNSS signals available, following a predefined trajectory with a total length of 9 m. The same three case studies were considered, namely event-only case, frames-only case, and combined case. Figure 11, Figure 12 and Figure 13 visually depict feature detection and tracking in each case. The resulting trajectories were then compared to the ground truth, as illustrated in Figure 14. Table 4 provides error statistics for the position in the three navigation solutions, while Table 5 shows the loop closing error, calculated as the difference in coordinates between the first and last points.
In the event-only case, the solution exhibited some drift after a short duration. The total RMSE for the event-only case was 0.55 m, with a loop closing error of 0.57 m. Conversely, the standard frame-only solution sustained a relatively small error over a more extended period, drifting by 0.41 m by the end of the trajectory. Its overall RMSE was 0.33 m. Notably, the combined solution yielded significantly fewer errors compared to the individual cases. The positioning RMSE decreased by 89% and 82%, compared to the event-only and standard frame-only cases, respectively. The loop closing error of the combined case was 0.04 m for a 9 m trajectory, indicating an accumulated drift of less than 1% compared to 6.3% and 4.5% in event-only and standard frame-only cases, respectively.

4. Conclusions

In this study, an event-based visual–inertial odometry algorithm for event cameras was developed for indoor navigation. The proposed approach consisted of two main threads: the front-end and the back-end. The front-end utilized the FAST corner detector for feature detection and the Lukas–Kanade tracking for feature tracking. The back-end, inspired by OKVIS, integrated events, frames, and inertial measurements to estimate the sensor state over time. The performance of the proposed approach was evaluated using simulated and real-world datasets. On simulated data, our approach outperformed U-SLAM, reducing mean positional error and RMSE by up to 50% and 47% on the x- and y-axes, respectively, with smaller maximum error. It also demonstrated real-time feasibility with 5–10 ms latency per event batch and 10–20 ms for frame updates, making it scalable for resource-constrained platforms such as UAVs.
For real-world DAVIS346-based datasets, three case studies were considered: event-only, frame-only, and combined cases. Two scenarios were considered: a ground-based dataset captured via manual navigation and a UAV-based dataset recorded with a quadrotor platform. The event-only scenario showed drift after a short duration, while the frame-only solution performed better. The combined case achieved substantial improvements in pose estimation, with RMSE reductions of 84% and 75% in the ground-based dataset and 89% and 82% in the UAV-based dataset, relative to event-only and frame-only cases, respectively. The accumulated drift of the combined case was under 1%. These results confirm that the proposed approach provides efficient and accurate indoor navigation, balancing performance with computational demands, and supporting real-time deployment on embedded systems.

Author Contributions

Conceptualization and methodology: A.E. and A.E.-R.; software: A.E.; validation: A.E.; formal analysis: A.E.; writing—original draft preparation, A.E.; writing—review and editing: A.E.-R.; supervision, A.E.-R. and S.J. All authors have read and agreed to the published version of the manuscript.

Funding

This research is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) RGPIN-2022-03822, the Government of Ontario, Toronto Metropolitan University, and SOTI Inc.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are not publicly available.

Acknowledgments

The authors would like to thank iniVation for making its DV packages available.

Conflicts of Interest

Author Sunil Jacob was employed by the company SOTI Aerospace, SOTI Inc. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

  1. Mazhar, F.; Khan, M.G.; Sällberg, B. Precise Indoor Positioning Using UWB: A Review of Methods, Algorithms and Implementations. Wirel. Pers. Commun. 2017, 97, 4467–4491. [Google Scholar] [CrossRef]
  2. Tiemann, J.; Ramsey, A.; Wietfeld, C. Enhanced UAV Indoor Navigation through SLAM-Augmented UWB Localization. In Proceedings of the 2018 IEEE International Conference on Communications Workshops (ICC Workshops), Kansas City, MO, USA, 20–24 May 2018; pp. 1–6. [Google Scholar]
  3. Tiemann, J.; Wietfeld, C. Scalable and precise multi-UAV indoor navigation using TDOA-based UWB localization. In Proceedings of the 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Sapporo, Japan, 18–21 September 2017; pp. 1–7. [Google Scholar]
  4. Raja, G.; Suresh, S.; Anbalagan, S.; Ganapathisubramaniyan, A.; Kumar, N. PFIN: An Efficient Particle Filter-Based Indoor Navigation Framework for UAVs. IEEE Trans. Veh. Technol. 2021, 70, 4984–4992. [Google Scholar] [CrossRef]
  5. Bachrach, A.; He, R.; Roy, N. Autonomous Flight in Unknown Indoor Environments. Int. J. Micro Air Veh. 2009, 1, 217–228. [Google Scholar] [CrossRef]
  6. Zhang, J.; Singh, S. LOAM: Lidar odometry and mapping in real-time. In Proceedings of the Robotics: Science and Systems, Berkeley, CA, USA, 12–16 July 2014; pp. 1–9. [Google Scholar]
  7. Yi, S.; Lyu, Y.; Hua, L.; Pan, Q.; Zhao, C. Light-LOAM: A Lightweight LiDAR Odometry and Mapping Based on Graph-Matching. IEEE Robot. Autom. Lett. 2024, 9, 3219–3226. [Google Scholar] [CrossRef]
  8. Bry, A.; Bachrach, A.; Roy, N. State estimation for aggressive flight in GPS-denied environments using onboard sensing. In Proceedings of the 2012 IEEE International Conference on Robotics and Automation, Saint Paul, MN, USA, 14–18 May 2012; pp. 1–8. [Google Scholar]
  9. Kumar, G.A.; Patil, A.K.; Patil, R.; Park, S.S.; Chai, Y.H. A LiDAR and IMU Integrated Indoor Navigation System for UAVs and Its Application in Real-Time Pipeline Classification. Sensors 2017, 17, 1268. [Google Scholar] [CrossRef]
  10. Hui, C.; Yousheng, C.; Shing, W.W. Trajectory tracking and formation flight of autonomous UAVs in GPS-denied environments using onboard sensing. In Proceedings of the Proceedings of 2014 IEEE Chinese Guidance, Navigation and Control Conference, Yantai, China, 8–10 August 2014; pp. 2639–2645. [Google Scholar]
  11. Qu, Y.; Yang, M.; Zhang, J.; Xie, W.; Qiang, B.; Chen, J. An Outline of Multi-Sensor Fusion Methods for Mobile Agents Indoor Navigation. Sensors 2021, 21, 1605. [Google Scholar] [CrossRef] [PubMed]
  12. Engel, J.; Schöps, T.; Cremers, D. LSD-SLAM: Large-Scale Direct Monocular SLAM. In Computer Vision—ECCV 2014; Springer International Publishing: Cham, Switzerland, 2014; pp. 834–849. [Google Scholar]
  13. Mur-Artal, R.; Montiel, J.M.M.; Tardos, J.D. ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Trans. Robot. 2015, 31, 1147–1163. [Google Scholar] [CrossRef]
  14. Klein, G.; Murray, D. Parallel Tracking and Mapping for Small AR Workspaces. In Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Japan, 13–16 November 2007; pp. 225–234. [Google Scholar]
  15. Campos, C.; Elvira, R.; Rodríguez, J.J.G.; Montiel, J.M.M.; Tardós, J.D. ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM. IEEE Trans. Robot. 2021, 37, 1874–1890. [Google Scholar] [CrossRef]
  16. Forster, C.; Zhang, Z.; Gassner, M.; Werlberger, M.; Scaramuzza, D. SVO: Semidirect Visual Odometry for Monocular and Multicamera Systems. IEEE Trans. Robot. 2017, 33, 249–265. [Google Scholar] [CrossRef]
  17. Taketomi, T.; Uchiyama, H.; Ikeda, S. Visual SLAM algorithms: A survey from 2010 to 2016. IPSJ Trans. Comput. Vis. Appl. 2017, 9, 16. [Google Scholar] [CrossRef]
  18. Endres, F.; Hess, J.; Sturm, J.; Cremers, D.; Burgard, W. 3-D Mapping With an RGB-D Camera. IEEE Trans. Robot. 2014, 30, 177–187. [Google Scholar] [CrossRef]
  19. Labbé, M.; Michaud, F. RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. J. Field Robot. 2019, 36, 416–446. [Google Scholar] [CrossRef]
  20. Kerl, C.; Sturm, J.; Cremers, D. Dense visual SLAM for RGB-D cameras. In Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, 3–7 November 2013; pp. 2100–2106. [Google Scholar]
  21. Gallego, G.; Delbruck, T.; Orchard, G.; Bartolozzi, C.; Taba, B.; Censi, A.; Leutenegger, S.; Davison, A.J.; Conradt, J.; Daniilidis, K.; et al. Event-Based Vision: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 44, 154–180. [Google Scholar] [CrossRef]
  22. Kim, H.; Leutenegger, S.; Davison, A. Real-time 3D reconstruction and 6-DoF tracking with an event camera. In Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; pp. 349–364. [Google Scholar] [CrossRef]
  23. Rebecq, H.; Horstschaefer, T.; Gallego, G.; Scaramuzza, D. EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real Time. IEEE Robot. Autom. Lett. 2017, 2, 593–600. [Google Scholar] [CrossRef]
  24. Delaune, J.; Bayard, D.S.; Brockers, R. Range-Visual-Inertial Odometry: Scale Observability Without Excitation. IEEE Robot. Autom. Lett. 2021, 6, 2421–2428. [Google Scholar] [CrossRef]
  25. Lee, M.S.; Jung, J.H.; Kim, Y.J.; Park, C.G. Event-and Frame-Based Visual-Inertial Odometry With Adaptive Filtering Based on 8-DOF Warping Uncertainty. IEEE Robot. Autom. Lett. 2024, 9, 1003–1010. [Google Scholar] [CrossRef]
  26. Mueggler, E.; Gallego, G.; Rebecq, H.; Scaramuzza, D. Continuous-Time Visual-Inertial Odometry for Event Cameras. IEEE Trans. Robot. 2018, 34, 1425–1440. [Google Scholar] [CrossRef]
  27. Zhu, A.Z.; Atanasov, N.; Daniilidis, K. Event-Based Visual Inertial Odometry. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 5816–5824. [Google Scholar]
  28. Rebecq, H.; Horstschaefer, T.; Scaramuzza, D. Real-time Visual-Inertial Odometry for Event Cameras using Keyframe-based Nonlinear Optimization. In Proceedings of the British Machine Vision Conference, London, UK, 4–7 September 2017. [Google Scholar]
  29. Zhou, Y.; Gallego, G.; Shen, S. Event-Based Stereo Visual Odometry. IEEE Trans. Robot. 2021, 37, 1433–1450. [Google Scholar] [CrossRef]
  30. Liu, Z.; Shi, D.; Li, R.; Yang, S. ESVIO: Event-Based Stereo Visual-Inertial Odometry. Sensors 2023, 23, 1998. [Google Scholar] [CrossRef] [PubMed]
  31. Rosten, E.; Drummond, T. Machine Learning for High-Speed Corner Detection. In Computer Vision—ECCV 2006; Springer: Berlin/Heidelberg, Germany, 2006; Volume 3951, pp. 430–443. [Google Scholar]
  32. Harris, C.; Stephens, M. A combined corner and edge detector. In Proceedings of the Alvey vision conference, Manchester, UK, 31 August–2 September 1988. [Google Scholar]
  33. Vasco, V.; Glover, A.; Bartolozzi, C. Fast event-based Harris corner detection exploiting the advantages of event-driven cameras. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Republic of Korea, 9–14 October 2016. [Google Scholar]
  34. Muggler, E.; Bartolozzi, C.; Scaramuzza, D. Fast Event-based Corner Detection. In Proceedings of the British Machine Vision Conference, London, UK, 4–7 September 2017. [Google Scholar]
  35. Chaudhry, R.; Ravichandran, A.; Hager, G.; Vidal, R. Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009. [Google Scholar]
  36. Clady, X.; Maro, J.-M.; Barré, S.; Benosman, R.B. A Motion-Based Feature for Event-Based Pattern Recognition. Front. Neurosci. 2016, 10, 594. [Google Scholar] [CrossRef]
  37. Lagorce, X.; Meyer, C.; Ieng, S.-H.; Filliat, D.; Benosman, R. Asynchronous Event-Based Multikernel Algorithm for High-Speed Visual Features Tracking. IEEE Trans. Neural Netw. Learn. Syst. 2015, 26, 1710–1720. [Google Scholar] [CrossRef]
  38. Zhu, A.Z.; Atanasov, N.; Daniilidis, K. Event-based feature tracking with probabilistic data association. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–03 June 2017. [Google Scholar]
  39. Besl, P.J.; McKay, N.D. A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 1992, 14, 239–256. [Google Scholar] [CrossRef]
  40. Gehrig, D.; Rebecq, H.; Gallego, G.; Scaramuzza, D. EKLT: Asynchronous Photometric Feature Tracking Using Events and Frames. Int. J. Comput. Vis. 2020, 128, 601–618. [Google Scholar] [CrossRef]
  41. Brandli, C.; Berner, R.; Yang, M.; Liu, S.-C.; Delbruck, T. A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor. IEEE J. Solid-State Circuits 2014, 49, 2333–2341. [Google Scholar] [CrossRef]
  42. Kueng, B.; Mueggler, E.; Gallego, G.; Scaramuzza, D. Low-latency visual odometry using event-based feature tracks. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Republic of Korea, 9–14 October 2016. [Google Scholar]
  43. Tedaldi, D.; Gallego, G.; Mueggler, E.; Scaramuzza, D. Feature detection and tracking with the dynamic and active-pixel vision sensor (DAVIS). In Proceedings of the 2016 Second International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP), Krakow, Poland, 13–15 June 2016. [Google Scholar]
  44. Vidal, A.R.; Rebecq, H.; Horstschaefer, T.; Scaramuzza, D. Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High-Speed Scenarios. IEEE Robot. Autom. Lett. 2018, 3, 994–1001. [Google Scholar] [CrossRef]
  45. iniVation. Dv-Mono-Vio-Sample. 2022. Available online: https://gitlab.com/inivation/dv/dv-mono-vio-sample (accessed on 21 March 2022).
  46. Lucus, B. An iterative registration technique with an application to stereo vision. In Proceedings of the Proceedings of the International Joint Conference on Artificial Intelligence, Vancouver, BC, Canada, 24–28 August 1981. [Google Scholar]
  47. Troiani, C.; Martinelli, A.; Laugier, C.; Scaramuzza, D. 2-Point-based Outlier Rejection for Camera-Imu Systems with applications to Micro Aerial Vehicles. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 31 May 2014. [Google Scholar] [CrossRef]
  48. Leutenegger, S.; Furgale, P.; Rabaud, V.; Chli, M.; Konolige, K.; Siegwart, R. Keyframe-based visual-inertial SLAM using nonlinear optimization. In Proceedings of the Robotis Science and Systems (RSS) 2013; Robotics: Science and Systems, Munich, Germany, 24–28 June 2013. [Google Scholar] [CrossRef]
  49. Strasdat, H.; Montiel, J.M.M.; Davison, A.J. Real-time monocular SLAM: Why filter? In Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, AK, USA, 3–7 May 2010; pp. 2657–2664. [Google Scholar]
  50. Agarwal, S.; Mierle, K. Ceres Solver. 2022. Available online: http://ceres-solver.org (accessed on 13 May 2022).
  51. Forster, C.; Carlone, L.; Dellaert, F.; Scaramuzza, D. On-Manifold Preintegration for Real-Time Visual--Inertial Odometry. IEEE Trans. Robot. 2017, 33, 1–21. [Google Scholar] [CrossRef]
  52. iniVation. DAVIS 346. 2022. Available online: https://inivation.com/wp-content/uploads/2019/08/DAVIS346.pdf (accessed on 7 April 2022).
  53. iniVation. DV Software. 2022. Available online: https://inivation.gitlab.io/dv/dv-docs/docs/tutorial-calibration/ (accessed on 17 March 2022).
  54. Meier, L.; Tanskanen, P.; Heng, L.; Lee, G.H.; Fraundorfer, F.; Pollefeys, M.J.A.R. PIXHAWK: A micro aerial vehicle design for autonomous flight using onboard computer vision. Auton. Robot. 2012, 33, 21–39. [Google Scholar] [CrossRef]
Sensors 25 00061 g001
Figure 1. Workflow of the proposed event-based VIO.
Figure 1. Workflow of the proposed event-based VIO.
Sensors 25 00061 g001
Sensors 25 00061 g002
Figure 2. DAVIS346 event camera.
Figure 2. DAVIS346 event camera.
Sensors 25 00061 g002
Sensors 25 00061 g003
Figure 3. Study area camera calibration: (a) a 6 × 9 chessboard with a square size of 30 mm; and (b) an example of the detected pattern image.
Figure 3. Study area camera calibration: (a) a 6 × 9 chessboard with a square size of 30 mm; and (b) an example of the detected pattern image.
Sensors 25 00061 g003
Sensors 25 00061 g004
Figure 4. An office environment simulation layout.
Figure 4. An office environment simulation layout.
Sensors 25 00061 g004
Sensors 25 00061 g005
Figure 5. Simulated dataset: comparison of trajectories.
Figure 5. Simulated dataset: comparison of trajectories.
Sensors 25 00061 g005
Sensors 25 00061 g006
Figure 6. Ground-based dataset: an example of feature detection and tracking on an events accumulated frame.
Figure 6. Ground-based dataset: an example of feature detection and tracking on an events accumulated frame.
Sensors 25 00061 g006
Sensors 25 00061 g007
Figure 7. Ground-based dataset: an example of feature detection and tracking on a standard frame.
Figure 7. Ground-based dataset: an example of feature detection and tracking on a standard frame.
Sensors 25 00061 g007
Sensors 25 00061 g008
Figure 8. Ground-based dataset: an example of feature detection and tracking on combined events and standard frames.
Figure 8. Ground-based dataset: an example of feature detection and tracking on combined events and standard frames.
Sensors 25 00061 g008
Sensors 25 00061 g009
Figure 9. Ground-based dataset: comparison of trajectories.
Figure 9. Ground-based dataset: comparison of trajectories.
Sensors 25 00061 g009
Sensors 25 00061 g010
Figure 10. UAV used for the experiments. (1) DAVIS346 event camera. (2) NVIDIA Jetson Xavier computer. (3) Pixhawk 4 flight controller.
Figure 10. UAV used for the experiments. (1) DAVIS346 event camera. (2) NVIDIA Jetson Xavier computer. (3) Pixhawk 4 flight controller.
Sensors 25 00061 g010
Sensors 25 00061 g011
Figure 11. UAV-based dataset: an example of feature detection and tracking on an events accumulated frame.
Figure 11. UAV-based dataset: an example of feature detection and tracking on an events accumulated frame.
Sensors 25 00061 g011
Sensors 25 00061 g012
Figure 12. UAV-based dataset: an example of feature detection and tracking on a standard frame.
Figure 12. UAV-based dataset: an example of feature detection and tracking on a standard frame.
Sensors 25 00061 g012
Sensors 25 00061 g013
Figure 13. UAV-based dataset: an example of feature detection and tracking on combined events and standard frames.
Figure 13. UAV-based dataset: an example of feature detection and tracking on combined events and standard frames.
Sensors 25 00061 g013
Sensors 25 00061 g014
Figure 14. UAV-based dataset: comparison of trajectories.
Figure 14. UAV-based dataset: comparison of trajectories.
Sensors 25 00061 g014
Table 1. Simulated dataset: comparison of the three cases’ position (m) error statistics.
Table 1. Simulated dataset: comparison of the three cases’ position (m) error statistics.
U-SLAMProposed Approach
MeanRMSEMaxMeanRMSEMax
X0.180.250.330.090.180.28
Y0.190.240.420.10.190.38
Z0.350.400.640.10.260.27
Total 0.31 0.21
Table 2. Ground-based dataset: comparison of the three cases’ position (m) error statistics.
Table 2. Ground-based dataset: comparison of the three cases’ position (m) error statistics.
Event-Only CaseFrame-Only CaseCombined Case
MeanRMSEMaxMeanRMSEMaxMeanRMSEMax
X1.41.602.520.771.332.44−0.170.350.92
Y−0.711.704.12−0.100.591.62−0.050.130.27
Total 2.33 1.45 0.37
Table 3. Ground-based dataset: comparison of the three cases closer error (m).
Table 3. Ground-based dataset: comparison of the three cases closer error (m).
Event-Only CaseFrame-Only CaseCOMBINED CASE
X1.581.950.12
Y−4.13−1.62−0.06
Total4.422.540.13
Table 4. UAV-based dataset: comparison of position (m) error statistics.
Table 4. UAV-based dataset: comparison of position (m) error statistics.
Event-Only CaseFrame-Only CaseCombined Case
MeanRMSEMaxMeanRMSEMaxMeanRMSEMax
X0.140.460.650.140.310.59−0.010.040.14
Y−0.10.300.330.010.110.240.040.050.09
Total 0.55 0.33 0.06
Table 5. UAV-based dataset: comparison of closer error (m).
Table 5. UAV-based dataset: comparison of closer error (m).
Event-Only CaseFrame-Only CaseCombined Case
X0.300.19−0.04
Y−0.48−0.360.01
Total0.570.410.04
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Elamin, A.; El-Rabbany, A.; Jacob, S. Event-Based Visual/Inertial Odometry for UAV Indoor Navigation. Sensors 2025, 25, 61. https://doi.org/10.3390/s25010061

AMA Style

Elamin A, El-Rabbany A, Jacob S. Event-Based Visual/Inertial Odometry for UAV Indoor Navigation. Sensors. 2025; 25(1):61. https://doi.org/10.3390/s25010061

Chicago/Turabian Style

Elamin, Ahmed, Ahmed El-Rabbany, and Sunil Jacob. 2025. "Event-Based Visual/Inertial Odometry for UAV Indoor Navigation" Sensors 25, no. 1: 61. https://doi.org/10.3390/s25010061

APA Style

Elamin, A., El-Rabbany, A., & Jacob, S. (2025). Event-Based Visual/Inertial Odometry for UAV Indoor Navigation. Sensors, 25(1), 61. https://doi.org/10.3390/s25010061

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop