Question about point-wise NCC vs. plane-induced patch NCC in CUDA MeshRefine

[will-zzy]

Hello @cdcseacave ,

Thank you for your excellent work on OpenMVS. I noticed that a CUDA version of MeshRefine has recently been introduced, and I would like to ask a technical question about the photometric consistency formulation.

From what I have observed, many recent mesh refinement methods seem to use a point-wise NCC formulation: for each reference pixel, the corresponding 3D point is obtained from the reference depth, projected into the source views, and the sampled source intensities are used to form an aligned image for computing NCC with the reference image.

In contrast, they appear to avoid the plane-induced patch NCC formulation commonly used in traditional MVS, where each reference pixel estimates a local plane from its normal and depth, derives a homography, warps the neighboring patch into the source view, and lets the NCC loss backpropagate through the homography to the mesh vertices.

May I ask whether point-wise NCC is generally considered the preferred or more stable choice for mesh refinement? I have implemented a preliminary mesh refinement variant based on plane-induced patch NCC, but the results were not as good as expected. I am wondering whether this is due to known stability issues of patch NCC in mesh refinement, or whether there are implementation details that are especially important for making this formulation work.

Any insights would be greatly appreciated.

[cdcseacave]

Plane-induced patch NCC is an approximation what works well in practice when the local surface is smooth; ideally if you can map pixel by pixel that is obviously more accurate in general. When refining meshes, you should use the knowledge on the available continuous surface (compared to point clouds that have no continuous surface knowledge) and that is what the mesh refinement implemented in OpenMVS is trying to do. How do you see the mesh refinement process?

[will-zzy]

I agree that, following the surface evolution formulation in [1], a pixel-to-pixel mapping should indeed be more accurate.

In my previous implementation, I used rasterized depth and face normals to perform plane-induced patch NCC optimization. However, for coarse-resolution meshes, the local planar approximation can introduce a large bias, which may explain why the optimization became worse instead of improving the geometry (left init mesh, right refined mesh by patch NCC).

When using point-wise NCC based on pixel-to-pixel mapping, I observed two issues. First, mesh refinement may cause a certain degree of surface erosion. For example, the wooden boards on the truck railing become thinner as optimization proceeds. This is relatively easy for me to understand: during the warping process, if there is no corresponding surface segment in space, reducing the NCC loss may naturally favor shrinking or removing that part of the surface.

Second, in some regions, mesh refinement makes the surface less smooth, as shown in the red box. My current interpretation is that this may be caused by repeated textures or weak textures. During refinement, the NCC loss moves vertices so that the warped patterns between views become more consistent. This is why Gaussian blur and image gradients are often used to provide a stable direction for pattern movement, as illustrated in the figure. However, repeated textures/weak textures can make the matching ambiguous and provide a noisy or uncertain gradient. I would like to ask whether there are better ways to handle this issue, or whether this is an inherent limitation of such photometric refinement.

Thank you for the further discussion!

[1] Vu, Hoang-Hiep, et al. "High accuracy and visibility-consistent dense multiview stereo." IEEE transactions on pattern analysis and machine intelligence 34.5 (2011): 889-901.