Prior-Driven Enhancements in 3D Gaussian Splatting

Abstract

3D Gaussian Splatting (3DGS) is a state-of-the-art technique for 3D scene rendering, offering high efficiency and excellent visual quality. However, because 3DGS relies on an initial sparse point set from Structure-from-Motion (SfM) and view-dependent properties, it can suffer from geometric inaccuracies and visual artifacts, particularly in complex scenes. To address these challenges, we propose an improved 3DGS approach that regularizes the optimization process by integrating geometric priors, including surface normals and dense depth information. Surface normal regularization improves geometric consistency by aligning Gaussian covariance with local surface structures, while dense depth priors combined with an initial points from SfM enhance per-pixel depth estimation, increasing accuracy and reducing ambiguities. These enhancements enable robust handling of diverse and complex real-world scenarios, minimizing visual distortions and improving reconstruction quality across various environments. To validate our method, we evaluate it on challenging datasets, including street-view scenes and highly reflective environments, while testing it across multiple SfM pipelines. Our results demonstrate compatibility across diverse environments and highlight the robustness of our approach. Experimental findings further show that our method enhances geometric accuracy and visual quality, establishing a reliable solution for real-time 3D scene rendering in complex environments.

Method

The same images feed two branches. An SfM pipeline (COLMAP, SuperPoint + SuperGlue, or LoFTR) estimates the camera poses and the sparse points that initialize the Gaussians. A monocular network (Metric3D) predicts a surface normal and a depth for every pixel. Both priors enter the 3DGS optimization, which otherwise stays unchanged.

Objective

\mathcal{L} = (1-\lambda_1)\,\mathcal{L}_{\text{color}} + \lambda_1\,\mathcal{L}_{\text{D-SSIM}} + \color{#c0392b}{\lambda_2\,\mathcal{L}_{\text{normal}} + \lambda_3\,\mathcal{L}_{\text{depth}}}

Eq. (7). The first two terms are the original 3DGS losses, and the terms in red are ours.

Normal prior regularization

Geometry-aware initialization aligns the orientation and scale of each initial Gaussian with the predicted surface normals. The normal-consistency loss then aligns the orientation and scale of every Gaussian's covariance with the surface, flattening it along the normal (after Hwang et al., 2024):

\mathcal{L}_{\text{normal}} = \lambda\,\mathcal{L}_{\text{axis}} + (1-\lambda)\,\mathcal{L}_{\text{scale}}

\mathcal{L}_{\text{axis}} = \frac{1}{N}\sum_{i=1}^{N}\sum_{j\in\{0,1,2\}}\bigl|\tilde R^{(i)}_G[:,j]\cdot\mathbf n^{(i)}\bigr|

\mathcal{L}_{\text{scale}} = \frac{1}{N}\sum_{i=1}^{N}\sum_{j\in\{0,1,2\}}\tilde s^{(i)}_G[j]\,\bigl|\tilde R^{(i)}_G[:,j]\cdot\mathbf n^{(i)}\bigr|

Depth prior regularization

The predicted depth D_dense has an unknown scale. It is rescaled to match the sparse depth D_sparse, obtained by projecting the SfM points onto each image, which gives the guide D_guide. The depth D rendered by Gaussian splatting is supervised with an L1 loss:

\mathcal{L}_{\text{depth}} = \bigl\lVert D_{\text{guide}} - D \bigr\rVert_1

Dense depth refines per-pixel geometry where the sparse SfM points leave it ambiguous, such as in repetitive patterns or on low-texture surfaces.

Interactive: the normal-consistency loss

A 2D slice of one Gaussian lying on a surface with normal n. Rotate it and change its shape. The two terms follow Eqs. (5)–(6) for this Gaussian. Both reach their minimum when the Gaussian lies flat on the surface, with its thinnest axis along the normal.

Rotation of the Gaussian Scale along axis 2 (thickness)

–L_axis

–L_scale

In 3D, the normal can only be orthogonal to two of the three axes. The minimum therefore puts the smallest scale on the axis that points along the normal, and the Gaussian becomes a thin disk on the surface.

Data

We use two Tanks and Temples scenes, Train and Horse, and two scenes we captured ourselves. Both of our scenes have repetitive patterns, strong reflections or low-texture surfaces, which make feature matching and pose estimation hard.

Results

Dataset	Method	COLMAP				SP-SG				LoFTR
Dataset	Method	MRE↓	PSNR↑	SSIM↑	LPIPS↓	MRE↓	PSNR↑	SSIM↑	LPIPS↓	MRE↓	PSNR↑	SSIM↑	LPIPS↓
Train	3DGS	0.75	21.10	0.802	0.218	1.40	21.10	0.750	0.282	0.79	20.97	0.767	0.274
Train	Ours	0.75	21.97	0.799	0.252	1.40	21.32	0.749	0.287	0.79	21.11	0.759	0.291
Horse	3DGS	0.71	24.18	0.889	0.239	1.31	21.01	0.802	0.239	0.80	23.39	0.870	0.174
Horse	Ours	0.71	25.50	0.903	0.153	1.31	21.12	0.801	0.246	0.80	24.80	0.881	0.165
Parking lots	3DGS	invalid	–	–	–	1.37	28.08	0.828	0.424	0.64	29.33	0.842	0.410
Parking lots	Ours	invalid	–	–	–	1.37	28.37	0.829	0.427	0.64	29.07	0.840	0.414
Street-view	3DGS	invalid	–	–	–	1.09	18.33	0.468	0.410	0.56	22.28	0.729	0.331
Street-view	Ours	invalid	–	–	–	1.09	19.00	0.479	0.492	0.56	22.49	0.722	0.340

Table 1 of the paper. MRE is the mean reprojection error of the SfM point cloud. COLMAP fails to reconstruct the Parking lots and Street-view scenes.

PSNR gain over 3DGS (dB)

PSNR rises in 9 of the 10 valid settings, for example from 21.10 to 21.97 on Train and from 24.18 to 25.50 on Horse with COLMAP. The exception is Parking lots with LoFTR (29.33 → 29.07), where reflections of artificial light on the floor blur the priors.
SSIM stays comparable, and LPIPS rises in some cases. Low-quality normals and depths can oversmooth high-frequency detail such as grass, foliage, gravel and clouds, while scene structure, edges and object placement are preserved.
A lower MRE goes with a higher PSNR and SSIM, while its correlation with LPIPS is weak. COLMAP does well on feature-rich benchmarks but fails with wide baselines and reflective surfaces. SP-SG and LoFTR reconstruct every scene, which makes LoFTR a strong alternative where COLMAP fails.

Prior-Driven Enhancements in 3D Gaussian Splatting:
Normals and Depths Regularization

Abstract

Method

Objective

Normal prior regularization

Depth prior regularization

Interactive: the normal-consistency loss

Data

Parking lots · Ladybug6

Street-view · Mobile Mapping System

Results

PSNR gain over 3DGS (dB)

Interactive 3D Gaussians

Side-by-Side Comparison

BibTeX