• when you need a high-fidelity facial capture but don’t have access to a 4D capture rig
• when you need to capture 4D sequences simultaniously with the body mocap
• when you need to achieve maximal actor likeness
• when you need to generate training data of the actor’s face for a neural net application

• a performance sequence captured on a 2-camera HMC. The actor should have markers on their face
• a checkerboard calibration sequence captured before or after the shooting
• a set of FACS scans captured in a photogrammetry rig. The scans should be wrapped into a consistent topology and stabilized
• a daily scan that has the same set of markers that was used during the capture. The daily scan can be captured using an iPhone, a hand-held camera, or a photogrammetry rig

No. The facial hair will lead to bad geometry fit.

The pipeline is designed to work with any stereo-camera HMC with vertical stereo pair. We highly recommend using strobe lights.

So far it was tested with Technoprops, SDeviation, and Faceform HMC.

No, not yet. We are working on that. But converting the results into blendshape rig controls would lose a significant amount of animation nuances.

No. The output is a point-cache animation that is not editable. We are working on an editable solution.

We are working on a retargeting tool but it’s not yet available. If the actor and the character look alike, you can use a naive approach to transferring the animation deltas.

Yes. For now it can work with 2 cameras on an HMC or 2 static machine-vision cameras.

Yes. The ImageFacialWrapping node processes each frame independently. For each frame, we need to be able to match a neutral face with any extreme facial expression.

The markers are needed to help find the correspondences where the changes in skin appearance are too big.

We normally perform processing with a team of people, dedicated to specific processing steps: tracking, detection, wrapping.

The tracking person tracks up to 30 seconds a day for a single camera view. The detection person produces around 30 seconds of detection a day. The ImageFacialWrapping step takes 5 to 10 minutes per frame so we normally run it on multiple machines overnight.

Having this data, the size of the team, and the number of machines, you can calculate the required timing.

We don’t recommend using this pipeline for extreme scenes. The wrapping process will capture all the secondary dynamic effects of the skin. Such sequences are very hard to stabilize with the current set of tools.

No. The processing of 4D sequences is a computationally-intensive offline process.