Live Demo

Abstract

Multi-view 3D geometry networks offer a powerful prior but are prohibitively slow for real-time applications. We propose a novel way to adapt them for online use, enabling real-time 6-DoF pose tracking and online reconstruction of objects and scenes from monocular RGB videos.

Our method rapidly selects and manages a set of images as keyframes to map a scene or object via π3 [32] with full bidirectional attention. We then cache the global self-attention block’s key-value (KV) pairs and use them as the sole scene representation for online tracking. This allows for up to 15× speedup during inference without the fear of drift or catastrophic forgetting. Our caching strategy is model-agnostic and can be applied to other off-the-shelf multi-view networks without retraining.

We demonstrate KV-Tracker on both scene-level tracking and the more challenging task of on-the-fly object tracking and reconstruction without depth measurements or object priors. Experiments on the TUM RGB-D, 7-Scenes, Arctic and OnePose datasets show the strong performance of our system while maintaining high frame-rates up to ∼30 FPS.

Explainer

Demos

Runtime Analysis

Frames per second (FPS) throughput comparison: processing N frames with full bidirectional attention vs. processing a single query frame with KV-cache from N frames.

Citation

@InProceedings{Taher_2026_CVPR,
    author    = {Taher, Marwan and Alzugaray, Ignacio and Mazur, Kirill and Kong, Xin and Davison, Andrew},
    title     = {KV-Tracker: Real-Time Pose Tracking with Transformers},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {28990-28999}
}

Acknowledgements

We would like to thank Nicholas Fry and Shinjeong Kim for their help presenting the demo in person at CVPR. Research presented here has been supported by Dyson Technology Ltd.

The website template was borrowed from Mip-NeRF.