In the evolution of speech and multimodal AI, how to enable large models to “understand” sound more naturally and directly and perform complex reasoning has always been a core direction explored by the industry. Recently, The Chinese University of Hong Kong, Shenzhen, and Microsoft Research jointly proposed a new multimodal unified modeling architecture called ConvexLLM, offering a new technical approach for integrating speech and language.

Core Innovation: C-Gate Architecture and the “Convex Hull Mapping” Principle

Traditional speech interaction pipelines often need to first transcribe speech into text before handing it to a large language model for processing. This may not only cause information loss, but also increases latency from intermediate steps.

ConvexLLM, by contrast, uses an innovative C-Gate architecture to directly map audio information such as speech into the “convex hull” of the large language model’s original text word embeddings. Specifically, this architecture generates audio representations using non-negative weighted combinations of frozen word embeddings. In this way, sound signals can be directly connected to and leverage the vast knowledge and reasoning capabilities accumulated by the large model during pretraining without first being transcribed into text, thereby achieving unified modeling of speech and language.

Evaluation Performance and Generalization Ability: Leading on Benchmarks, Supporting Multiple Types of Audio Understanding

In terms of capability validation, ConvexLLM has demonstrated standout performance. In several benchmarks such as VoiceBench, this architecture achieved leading results among open-source models of comparable scale; on the BBH task, it exceeded the strongest baseline by 8.8 percentage points.

Beyond speech interaction, ConvexLLM also has strong adaptability across multiple scenarios, and can support audio understanding tasks for multiple types such as environmental sounds and music, further expanding the application boundaries of unified audio modeling.