The Ultimate Guide to Robot State Estimation (15)：VIO/VSLAM/Depth Camera

Within the 3D SLAM technology ecosystem, VIO-SLAM (Visual-Inertial Odometry-Simultaneous Localization and Mapping) faces long-standing core obstacles to engineering implementation compared to Lidar-SLAM and LIO-SLAM. These challenges include not only issues with algorithm accuracy itself but also critical problems such as insufficient robustness, scale inaccuracies, difficulties in constructing an “environmental perception” (or “worldview”), and high computational costs. Below is a systematic explanation focusing on accuracy-related questions, the design logic of fusion frontends, and the technical characteristics of mainstream depth cameras, providing references for engineering practice.

I. In-Depth Analysis of VIO & VSLAM Accuracy Issues

1. The Gap Between Academic Metrics and Engineering Practice

In academic research, RMSE (Root Mean Square Error) and ATE (Absolute Trajectory Error) are core metrics for evaluating system accuracy. However, it is important to note that most papers verify these metrics under “ideal experimental conditions.” For example, they rely on standardized datasets (with rigorously calibrated sensor parameters and controlled environmental variables), and the calibration and optimization of sensor groups are typically conducted by professional research teams—resulting in hardware combinations that already have high accuracy and stability.

In real-world engineering scenarios, factors such as fluctuations in sensor performance and environmental interference (e.g., sudden changes in lighting, occlusions) can lead to accuracy deviations. Therefore, academic metrics cannot be directly equated to real-world implementation results; secondary verification based on actual hardware and scenarios is essential.

2. Multi-Dimensional Core Factors Affecting Accuracy (Not Determined by a Single Optimization Method)

Accuracy is not a “mysterious concept” but is jointly determined by multiple factors across three dimensions: hardware, algorithms, and the environment. Specific factors include:

• Hardware Dimension: Sensor combination type (monocular/binocular camera, IMU performance level), shutter type (global shutter/rolling shutter), camera intrinsic/extrinsic calibration accuracy, IMU bias control effectiveness, and time synchronization (Td) accuracy between sensors;
• Algorithm Dimension: Frontend feature processing methods (direct method/feature point method), backend optimization type (BA—Bundle Adjustment/KF—Kalman Filter). While BA is traditionally perceived as more accurate, KF offers a better balance between “real-time performance and accuracy” in scenarios with limited computing power. Higher computing power input does not always guarantee higher accuracy;
• Environment Dimension: Operating scenario (texture-rich indoor environments/light-variable outdoor environments), stability of photometric changes (e.g., sudden shifts in strong light, shadow occlusions).

Additionally, all nonlinear optimization algorithms suffer from “long-term error accumulation.” Thus, loop closure detection is critical for long-term accuracy control—systems lacking loop closure capabilities can only be called “odometers,” not complete SLAM systems. During long-term operation, among the error controls for 3D rotation (attitude) and 3D translation (position), rotation errors have a more significant impact: minor attitude deviations amplify with movement distance, eventually leading to severe position inaccuracies.

II. Design Logic of Fusion Frontends for VIO & VSLAM

Since the introduction of the VINS-FUSION system, the importance of “multi-sensor fusion” in VIO has gradually become an industry consensus. VIO itself is a fusion of cameras and IMUs, and positioning modules such as GNSS/RTK are often integrated into the system architecture through loose coupling (e.g., as prior information or relative beacons) or tight coupling (e.g., the deep fusion solution in the GVINS system).

Frontend fusion (combining multiple algorithmic approaches) will become a mainstream technical direction in the future, but it must be based on “controllable computational costs.” Below is an analysis of current typical technical paths and core characteristics:

1. Comparison of Mainstream Fusion Frontend Solutions

• Semi-Direct Method + Feature Point Method: This solution balances the real-time performance of the direct method and the robustness of the feature point method, with full consideration of computational cost control during design—making it a classic early fusion frontend approach. However, its limitations include the lack of real-sensor optimization modules in open-source versions and a relatively sparse “environmental perception” (worldview), which makes it difficult to support mapping requirements in complex scenarios.
• Direct Method + ORB Feature Point Method: The direct method excels at using image gradient information for fast motion estimation, while the ORB feature point method improves feature matching stability through centroid calculation and descriptor generation. The two complement each other, enabling adaptation to more texture conditions. Nevertheless, this solution requires adjusting feature extraction parameters for different scenarios, and its overall adaptability still needs further optimization.
• SuperPoint + LK Optical Flow: SuperPoint offers high robustness and can adapt to complex photometric environments (e.g., sudden light changes, shadow coverage), while LK optical flow excels at capturing motion trajectories between consecutive frames for smooth inter-frame matching. However, this solution has obvious limitations: SuperPoint incurs high computational costs (only 70 frames per second at VGA resolution on a Titan X graphics card). When deployed on edge devices, a balance between “number of feature points” and “frame rate” must be struck, and the debugging and optimization process in open-source frameworks is relatively complex.

2. Core Design Logic of Fusion Frontends: Algorithmic Complementarity

The essence of a fusion frontend is “complementarity of advantages”—combining different algorithms to cover the needs of more scenarios:

• Feature point methods such as ORB and SuperPoint: Excel at extracting salient features from images and generating robust descriptors, delivering high matching accuracy in texture-rich scenarios;
• LK optical flow and direct methods: Better at using image gradient information to achieve motion estimation without relying on explicit feature points. They can effectively capture object contours between consecutive frames, making them suitable for low-texture or fast-motion scenarios.

Based on industry practice and technical exchanges (e.g., discussions with Dr. Xu Hao), the mainstream fusion direction in the future may be “Direct Method + SuperPoint,” with neither acting as a secondary component (loop closure detection is implemented based on SuperPoint). From the perspective of scientific logic and engineering feasibility, this combination can fully leverage the real-time performance of the direct method and the robustness of SuperPoint, offering high application value.

3. Reference for a Typical System: RoboBaton-VIOBOT2

When used solely as an odometer, RoboBaton-VIOBOT2 performs excellently in terms of “triple performance metrics” (computational cost, robustness, accuracy) and “worldview” construction. However, it still has room for optimization in scenarios with extreme photometric changes (e.g., sudden strong light) or extremely low illumination. When used as an odometer, it can be connected to RTK, wheel encoders, or other positioning devices to input absolute/relative beacon information, further improving operational stability.

III. Technical Characteristics and Engineering Adaptability of Mainstream Depth Cameras

Current mainstream depth cameras can be divided into two categories. Below are their technical principles, core characteristics, and limitations in engineering applications—providing references for hardware selection:

1. Active Projection Integral Depth Cameras

The technical principles of these cameras mainly include i-TOF (Time of Flight) and structured light. Their core advantage is the ability to output dense depth data, which can meet the needs of detailed mapping or object detection. However, from an engineering adaptability perspective, they have several limitations:

• Common Limitations: Processing dense point clouds requires extremely high computational power, resulting in low usability on edge devices (embedded devices). They are highly dependent on algorithms such as PCL (Point Cloud Library), PCA (Principal Component Analysis), and ICP (Iterative Closest Point). If downsampling is used to reduce costs, the core value of “density” is lost. Thus, they are only suitable for simple scenarios such as obstacle avoidance and end-stage operation correction, and cannot be used as complete SLAM frontends.
• Differentiated Limitations: They have a short detection range, making them mostly suitable for short-distance indoor scenarios. In outdoor environments, they are easily interfered with by factors such as direct sunlight, leading to significant declines in depth data accuracy. They also require support from high-performance ISPs (Image Signal Processors), and the computational costs of integration and filtering cannot be ignored.

2. Binocular Disparity Calculation Depth Cameras

These cameras obtain depth information based on disparity calculations from binocular cameras. Their technical principle is relatively straightforward (repeated disparity calculation logic), and they can adapt to different detection ranges by adjusting the binocular baseline. They also support GPU/NPU hardware acceleration, with dedicated computing support now available from ASIC chips such as Movidus and HiSilicon DV500. More related hardware solutions are expected to become available in the future.

Their limitations include the aforementioned “high computational cost of dense point clouds” (a common issue). Additionally, they suffer from the “hole effect”—missing depth data in areas with object occlusions or no texture. Currently, some manufacturers have partially addressed the “hole effect” by “fusing visible light texture features (real-time synchronized dual-light data) + object detection algorithms.” However, this implementation is complex and still fails to overcome the core limitation of “inability to support complete SLAM on edge devices.”

Conclusion  

The engineering implementation of VIO and VSLAM requires striking a balance between “accuracy control, computational cost, and hardware adaptability.” The key lies in the algorithm selection for fusion frontends and the scenario matching of depth cameras. It is necessary to develop optimal solutions between “performance indicators” and “costs” based on specific application requirements.