MotionCtrl Model Card

Project Page | Paper (ArXiv) | Code

🤗 Gradio demo (MotionCtrl+VideoCrafter) | 🤗 Gradio demo (MotionCtrl+SVD)

Introduction

MotionCtrl, when provided with specific camera movements indicated by camera poses or several trajectories, can independently control both the camera and object motion in a generated video. It can be deployed on both Text-to-Video generation models, such as LVDM/VideoCrafter and AnimateDiff, as well as Image-to-Video generation models like SVD.

MotionCtrl + SVD

• Model: motionctrl.pth
• Results:

 

MotionCtrl + LVDM/VideoCrafter

• Model:
  • motionctrl_svd.ckpt
  • motionctrl_videocrafter2_cmcm.ckpt
• Results:

 

MotionCtrl + AnimateDiff

• Model:
  • motionctrl_animatediff_cmcm.ckpt
  • motionctrl_animatediff_omcm.ckpt
• Results:

 

Usage

• Download directly in this repository OR Download with python script

from huggingface_hub import hf_hub_download
motionctrl_lvdm_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl.ckpt", repo_type="model")
motionctrl_svd_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl_svd.ckpt", repo_type="model")

• Then generate the controllable video following the instructions in our GitHub repository.

Citation

@inproceedings{wang2023motionctrl,
  title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation},
  author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin},
  booktitle={arXiv preprint arXiv:2312.03641},
  year={2023}
}