Lip Sync - Sonic

Aim

Came across this video on instagram and I wanted to re-create it - Gabri Omoniyi on Instagram

Implementation

I went down a rabbit hole of:

• running deep research tools
• asking questions on discord, reddit and X
• going through Chinese blogging websites

I worked with multiple platforms like Hedra, PikaLabs, Kling, etc but none gave satisfactory results - e.g -

I finally came across Sonic and the results matched what I wanted. What followed was a quick implementation in comfy_ui (luckily, someone had already made nodes for this: GitHub - smthemex/ComfyUI_Sonic).

Here are the results from the testing -

Parameters that worked for me -

• ip_audio_scale - 0.8 – 0.9
• fps - 25
• steps - 25 (the articles claim that the model is trained on this)
• fp16

I did find the original creator and reached out to him. They created the original video using - CapCut Pro. Since this is banned in India, I could not run tests on it.