Domain
Automotive, Robotics, Consumer Electronics
Challenge
Synchronize and fuse LiDAR and MIPI camera data in real time, enabling accurate timestamp alignment and point cloud visualization across diverse perception applications.
Solution
A modular GStreamer-based sensor fusion framework plugin engineered by MosChip, integrating LiDAR and camera streams with precise synchronization, SEI data embedding, and real-time point cloud overlay.
Platform
Xilinx UltraScale+ MPSoC, GStreamer, Ubuntu/Linux, C
MosChip designed a sensor fusion platform for use cases such as Automotive ADAS, Robotics, environmental mapping, video monitoring and control, and advanced content streaming domains. The solution was built to accurately synchronize LiDAR and MIPI camera inputs, overcoming challenges associated with varying sensor frequencies while delivering precise perception data alignment for both edge and cloud-based applications.
MosChip developed a portable GStreamer-based framework with custom plugins for LiDAR data capture, timestamp synchronization, SEI-based embedding and extraction of LiDAR point cloud data within H.264 and H.265 video streams, and real-time overlay of LiDAR data onto live video feeds. Built on the Xilinx UltraScale+ platform, the framework was designed for portability and scalability, enabling seamless integration with different sensors, codecs, and deployment environments.
The result was a reusable and high-performance sensor fusion platform that eliminated data synchronization mismatches, enabled accurate depth perception and terrain mapping, reduced integration effort through reusable building blocks, and accelerated deployment across automotive, robotics, multimedia, and smart vision applications.
The core challenge was to design and implement a solution for capturing and synchronizing data from a MIPI camera, a LiDAR sensor, and multiple additional sensors simultaneously. The use case spanned depth acquisition, LiDAR-based terrain mapping, and incremental ADAS enhancements, each placing different demands on the data pipeline.
To make this work across all these applications, the solution needed to address three specific technical requirements:
The challenge was not just technical depth in each of these areas individually. It was building a solution that handled all three reliably and continuously.
MosChip developed a GStreamer-based sensor fusion solution that can be extended with various sensors, including LiDAR and MIPI, as well as different video encoders. The architecture was built for portability and reuse, making it straightforward to add new sensor types or adapt the pipeline for different deployment environments. The key elements of the solution included:
| Component | Specification |
|---|---|
| HardwarePlatform/SoC | Xilinx UltraScale+ MPSoC |
| Codecs | H.264/H.265 Encoder and Decoder |
| Stream Resolution | 4Kp30 |
| LiDAR ScanFrequency | 12Hz scan frequency, 5000Hz range frequency |
| Framework | GStreamer |
| Operating System | Ubuntu/Linux |
| Programming Language | C |
By building on the GStreamer ecosystem and designing for modularity from the ground up, MosChip delivered a solution that goes beyond solving the immediate sensor fusion problem. The architecture was built to keep delivering value over time as requirements evolve and new sensors are added.
LiDAR and camera sensor fusion combines depth information from LiDAR with visual data from cameras to create a more accurate understanding of the surrounding environment. By leveraging the strengths of both sensors, it improves object detection, environmental perception, and decision-making for applications such as ADAS, robotics, autonomous systems, and industrial automation.
Accurate synchronization ensures that data captured by different sensors represents the same moment in time. Proper timestamp alignment helps prevent data mismatches, improves sensor fusion accuracy, and enables reliable perception, tracking, and real-time decision-making in safety-critical and autonomous applications.
A modular sensor fusion architecture makes it easier to integrate new sensors, support different hardware platforms, and adapt to evolving application requirements. It reduces development effort, improves reusability, accelerates deployment, and provides a scalable foundation for future perception systems.
GStreamer provides a flexible multimedia framework for capturing, processing, synchronizing, and streaming data from multiple sensors in real time. Its plugin-based architecture enables developers to build scalable pipelines for applications such as video analytics, robotics, ADAS, industrial vision, and edge AI systems.
A mid-sized construction company utilizes a 250 KW generator set to power various operations including tower cranes, project offices, labour accommodation, lighting, and….
The client is a leading US-based computing solutions company providing solutions for SoC-based microcontrollers, catering to defense, automotive, and industrial applications. …
The client is a leading US-based computer hardware manufacturing company having expertise in CPU development involving the latest technologically advanced methodologies…
Join us at India’s premier semiconductor ecosystem event – SEMICON India 2026 to explore our comprehensive...
How can manufacturers transform fragmented operational data into real-time intelligence...
Today’s consumer products respond after something has happened. Your wearable tells you your heart rate is elevated.…
Recently, AI processors have seen significant improvements in computing power due to specialized units such as NPUs…
