MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices

Kejie Li¹ Jia-Wang Bian¹, Robert Castle², Philip H.S. Torr¹, Victor Adrian Prisacariu¹
¹ University of Oxford ² Apple

Abstract

Generate high-quality 3D ground-truth shapes for reconstruction evaluation is extremely challenging because even 3D scanners can only generate pseudo ground-truth shapes with artefacts. We propose a novel data capturing and 3D annotation pipeline to obtain precise 3D ground-truth shapes without relying on expensive 3D scanners. The key to creating the precise 3D ground-truth shapes is using LEGO models, which are made of LEGO bricks with known geometry. The MobileBrick dataset provides a unique opportunity for future research on high-quality 3D reconstruction thanks to two distinctive features: 1) A large number of RGBD sequences with precise 3D ground-truth annotations. 2) The RGBD images were captured using mobile devices so algorithms can be tested in a realistic setup for mobile AR applications.

3D Object Alignment

We showcase the quality of GT shape alignment to the image sequence by projecting the 3D model to the RGB images (shown as "3D Model Projection"). "GT depth" is the depth maps rendered from 3D model at the same viewpoint. "RGB Image" and "ARKit Depth" are the high-res RGB images and low-res depth maps provided by ARKit.

Benchmark Results

The methods achiving the best and second best performance are denoted in Green and Yellow respectively
Methods	Input Data	σ =2.5mm			σ =5mm			Chamfer (mm)
Methods	Input Data	Accu. (%)	Rec. (%)	F1	Accu. (%)	Rec. (%)	F1	Chamfer (mm)
TSDF-Fusion	Depth	42.07	22.21	28.77	73.46	42.75	53.29	13.78
BNV-Fusion	Depth	41.77	25.96	33.27	71.20	47.09	55.11	9.60
Neural-RGBD	RGBD	20.61	10.66	13.67	39.62	22.06	27.66	22.78
COLMAP	RGB	74.89	68.20	71.08	93.79	84.53	88.71	5.26
Vis-MVSNet	RGB	79.83	47.25	58.32	97.35	65.90	77.49	9.27
Vis-MVSNet (finetuned)	RGB	75.64	53.64	62.01	96.03	72.42	81.89	9.52
NeRF	RGB	47.11	40.86	43.55	78.07	69.03	73.45	7.98
NeuS	RGB	77.35	70.85	73.74	93.33	86.11	89.30	4.74
Voxurf	RGB	78.44	72.41	75.13	93.95	87.53	90.41	4.71

BibTeX

    @article{li2023mobilebrick,
      author = {Kejie Li, Jia-Wang Bian, Robert Castle, Philip H.S. Torr, Victor Adrian Prisacariu},
      title = {MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices},
      journal={arXiv preprint arXiv:2303.01932},
      year={2023}
    }