The use of motion signals is a novel technique, which the company is calling omni-conditions training. With this, the AI model is trained on different modalities, including text, image, audio, and video. Researchers said this allowed the model to learn mixed conditioning which overcame the scarcity of high-quality data.

OmniHuman can turn photos into realistic videos of people speaking, singing and moving naturally, based on 18,700 hours of human motion data.