Technologies on Display

G4 - A Multilingual Virtual Performer

Virtual performers have been heavily used in computer games, TV programmes and movie production. However, lip synchronization is still a tedious and difficult task. Animators have to spend hours and hours of time to fine tune the parameters in order to achieve realistic effect. We developed a system that allows us to easily model virtual characters with realistic and natural talking and singing abilities. It provides natural human-computer interface. The system offers a real-time text-to-speech synthesis. Lip shape parameters are generated and fed to the rendering engine for facial animation in real-time. Since the system is based on International Phonetic Alphabet (IPA) standard, it supports multi-language including Cantonese, Mandarin, Japanese, and English. Hence it can be easily configured for both local and foreign markets. Besides realistic rendering, it also allows stylish rendering such as comic-style rendering. All these processing are done on an ordinary PC. With this system, animators can easily produce lip-synchronized character facial animation. There are a lot of applications including character modeling in computer games, TV and movies, computer-assisted education and even the latest virtual idol.


  • Digital entertainment
  • Realistic and natural character modeling in computer games and TV programme
  • Real-time Text-To-Speech synthesis and Lip shape parameter generation
  • Education and virtual agent


  • Real-time, realistic facial animation with emotional expression on ordinary PC
  • Stylish rendering such as comic style rendering
  • Real-time text-to-speech synthesis and lip shape parameters generation
  • Based on International Phonetic Alphabet (IPA) standard, brings us Multi-language support including Cantonese, Mandarin, Japanese, and English
  • Talking as well as singing


Principal Investigator
Prof. P. A. Heng
Department of Computer Science and Engineering