testing with our own data

[kinta0721]

Thank you for your impressive work!

I want to apply our data, multiple images (same img size as your project ) and same format calibration data(*.tka) used in your project ,to your test code with pre-trained model.
However the generated results, the generated mesh looks crash.
Is there anything I should be careful about?

Thanks

[TimoBolkart]

Thank you for your interest in TEMPEH. As TEMPEH is trained on a constrained multi-view face dataset with a fixed set of cameras (i.e., only with a small variation in camera intrinsics and extrinsics across captures), a model trained on such data is unlikely to generalize to entirely different captures. If you want to use the method on your own data, I recommend training (or at least fine-tuning the trained model) on parts of your data set.

[kinta0721]

Ikinta0721 commented now
I appreciate your response and advice.
I see, and I am curious about which part or parameter is strongly dependent on the trained dataset. (i.e., course mesh grid or camera angle or.....)
And, does it work for applying open datasets such as multiface released by Meta?

I would be happy if you could reply.
Thanks.

[TimoBolkart]

TEMPEH applies a U-Net type feature extractor network to each of the input images to extract feature maps. As the model is trained on a single dataset with similar views and camera parameters (i.e., camera intrinsics rarely change in a fixed multi-view setup) across captures, this feature extractor overfits to the used camera setting.

For training TEMPEH, we also use the grey-scale stereo images, which have a very specific structure. Applying the trained model to a different dataset such as multiface will therefore not result in good results. We did not try training on multiface data though.

[TimoBolkart]

Regarding your question @kinta0721, there is an ECCV 2024 paper GRAPE: Generalizable and Robust Multi-view
Facial Capture which extends TEMPEH to generalize across datasets. They find that replacing the spatial transformer for head localization by some visual hull head localization along with using background segmented training images allows a TEMPEH-type model to generalize across datasets.