Camera-based 3D object detection and tracking are central to autonomous driving, yet precise 3D object localization remains fundamentally constrained by depth ambiguity when no expensive, depth-rich online LiDAR is available at inference. In many deployments, however, vehicles repeatedly traverse the same environments, making static point cloud maps from prior traversals a practical source of geometric priors. We propose DualViewMapDet, a camera-only inference framework that retrieves such map priors online and leverages them to mitigate the absence of a LiDAR sensor during deployment. The key idea is a dual-space camera-map fusion strategy that avoids one-sided view conversion. Specifically, we (i) project the map into perspective view (PV) and encode multi-channel geometric cues to enrich image features and support BEV lifting, and (ii) encode the map directly in bird's-eye view (BEV) with a sparse voxel backbone and fuse it with lifted camera features in a shared metric space. Extensive evaluations on nuScenes and Argoverse 2 demonstrate consistent improvements over strong camera-only baselines, with particularly strong gains in object localization. Ablations further validate the contributions of PV/BEV fusion and prior-map coverage.

Overview of DualViewMapDet. Given multi-view RGB images and a local static point cloud map patch retrieved at the current ego location, we employ image-map grid masking during training to improve robustness (omitted in PV map encodings). The images are encoded into multi-scale PV features, while the map is encoded in perspective view by projecting it into each camera to form multi-channel PV map encodings and in bird’s-eye view into a BEV feature grid. We then fuse camera and map features in PV, lift the fused PV features to BEV, and fuse them with the BEV map features in a shared metric space. Finally, a transformer-based head with BEV-PV deformable aggregation decodes 3D bounding boxes from the fused PV and BEV representations.

Markus Käppeler, Özgün Çiçek, Yakov Miron, Abhinav Valada

@article{kaeppeler2026dvmd,
      author={Käppeler, Markus and Çiçek, Özgün and Miron, Yakov and Valada, Abhinav},
      title={Leveraging Previous-Traversal Point Cloud Map Priors for Camera-Based 3D Object Detection and Tracking}, 
      journal={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
      year={2026},
}