In a second of pure serendipity, Lah Yileh Lee and Xinting Lee, a pair of proficient singers who typically stream their performances on-line, discovered themselves performing in a public sq. in Taipei when NVIDIA founder and CEO Jensen Huang occurred upon them.
Huang couldn’t resist becoming a member of in, cheering on their serenade as they recorded Lady Gaga’s “Always Remember Us This Way.”
The ensuing video shortly went viral, as did a follow-up video from the pair, who sang Lady Gaga’s “Hold My Hand,” the music Huang initially requested.
Toy Jensen Created Utilizing NVIDIA Omniverse Avatar Cloud Engine
Now, with the assistance of his AI-driven avatar, Toy Jensen, Huang has provide you with a playful holiday-themed response.
NVIDIA’s artistic group shortly developed a vacation efficiency by TJ, a tech demo showcasing core applied sciences which can be a part of the NVIDIA Omniverse Avatar Cloud Engine, or ACE, platform.
Omniverse ACE is a group of cloud-native AI microservices and workflows for builders to simply construct, customise and deploy participating and interactive avatars.
Not like present avatar improvement, which requires experience, specialised tools, and manually intensive workflows, Omniverse ACE is constructed on high of the Omniverse platform and NVIDIA’s Unified Compute Framework, or UCF, which makes it attainable to shortly create and configure AI pipelines with minimal coding.
“It’s a very wonderful know-how, and the truth that we are able to do that is phenomenal,” mentioned Cyrus Hogg, an NVIDIA technical program supervisor.
To make it occur, NVIDIA’s group used a lately developed voice conversion mannequin to extract the voice of an expert singer from a pattern supplied by them and switch it into TJ’s voice – initially developed by coaching on hours of actual world recordings. They used the musical notes from that pattern and utilized them to the digital voice of TJ to make the avatar sing the identical notes and with the identical rhythm as the unique singer.
NVIDIA Omniverse Generative AI – Audio2Face, Audio2Gesture Allow Practical Facial Expressions, Physique Actions
Then the group used NVIDIA Omniverse ACE together with Omniverse Audio2Face and Audio2Gesture applied sciences to generate real looking facial expressions and physique actions for the animated efficiency based mostly on TJ’s audio alone.
Whereas the group behind Omniverse ACE applied sciences spent years growing and fine-tuning the know-how showcased within the efficiency, turning the music observe they created into a refined video took simply hours.
Toy Jensen Delights Followers With ‘Jingle Bells’ Efficiency
That gave them loads of time to make sure a tremendous efficiency.
They even collaborated with Jochem van der Saag, a composer and producer who has labored with Michael Bublé and David Foster, to create the proper backing observe for TJ to sing alongside to.
“We’ve got van der Saag composing the music, and he’s gonna additionally orchestrate it for us,” mentioned Hogg. “In order that’s a very welcome boost to the group. And we’re actually excited to have him on board.”
ACE Might Revolutionize Digital Experiences
The result’s the proper showcase for NVIDIA Omniverse ACE and the purposes it might have in numerous industries — for digital occasions, on-line training and customer support, in addition to in creating personalised avatars for video video games, social media and digital actuality experiences. NVIDIA Omniverse ACE might be out there quickly to early-access companions.
