G4 - A Multilingual Virtual Performer

Virtual performers have been heavily used in computer games, TV programmes and movie production. However, lip synchronization is still a tedious and difficult task. Animators have to spend hours and hours of time to fine tune the parameters in order to achieve realistic effect. We developed a system that allows us to easily model virtual characters with realistic and natural talking and singing abilities. It provides natural human-computer interface. The system offers a real-time text-to-speech synthesis. Lip shape parameters are generated and fed to the rendering engine for facial animation in real-time. Since the system is based on International Phonetic Alphabet (IPA) standard, it supports multi-language including Cantonese, Mandarin, Japanese, and English. Hence it can be easily configured for both local and foreign markets. Besides realistic rendering, it also allows stylish rendering such as comic-style rendering. All these processing are done on an ordinary PC. With this system, animators can easily produce lip-synchronized character facial animation. There are a lot of applications including character modeling in computer games, TV and movies, computer-assisted education and even the latest virtual idol.

Applications

• Digital entertainment
• Realistic and natural character modeling in computer games and TV programme
• Real-time Text-To-Speech synthesis and Lip shape parameter generation
• Education and virtual agent

Features

• Real-time, realistic facial animation with emotional expression on ordinary PC
• Stylish rendering such as comic style rendering
• Real-time text-to-speech synthesis and lip shape parameters generation
• Based on International Phonetic Alphabet (IPA) standard, brings us Multi-language support including Cantonese, Mandarin, Japanese, and English
• Talking as well as singing

Website
http://www.cse.cuhk.edu.hk/~crc

Principal Investigator
Prof. P. A. Heng
Department of Computer Science and Engineering