Abstract

The main challenges of 3D pose transfer are: 1) Lack of paired training data with different characters perform- ing the same pose; 2) Disentangling pose and shape infor- mation from the target mesh; 3) Difficulty in applying to meshes with different topologies. We thus propose a novel weakly-supervised keypoint-based framework to overcome these difficulties. Specifically, we use a topology-agnostic keypoint detector with inverse kinematics to compute trans- formations between the source and target meshes. Our method only requires supervision on the keypoints, can be applied to meshes with different topologies and is shape- invariant for the target which allows extraction of pose-only information from the target meshes without transferring shape information. We further design a cycle reconstruction to perform self-supervised pose transfer without the need for ground truth deformed mesh with the same pose and shape as the target and source, respectively. We evaluate our approach on benchmark human and animal datasets, where we achieve superior performance compared to the state-of-the-art unsupervised approaches and even compa- rable performance with the fully supervised approaches. We test on the more challenging Mixamo dataset to verify our approach’s ability in handling meshes with different topolo- gies and complex clothes. Cross-dataset evaluation further shows the strong generalization ability of our approach.

Method

The overall framework of our proposed approach. The left part is our pipeline for pose transfer, which contains four learnable components: a keypoints detection network, a twist prediction network, a skinning weights prediction network, and a refinement network, and two functions: an Inverse and Forward Kinematics function and an LBS function. The right part is an illustration of the cycle reconstruction process. The yellow and blue meshes represent two different characters.

Visualization

We show some results on stylized human meshes and animal meshes.

Comparison

We show comparison with SKF (ECCV 2022 paper Skeleton-free Pose Transfer for Stylized 3D Characters).

@article{jnchen23ws3dpt,
                title={Weakly-supervised 3D Pose Transfer with Keypoints},
                author={Jinnan Chen and Chen Li Gim Hee Lee},
                year={2023}
}