Recently, the Tencent Hunyuan team announced a new technical achievement, successfully compressing its 1.8B translation model, which is mainly for edge devices, to a few hundred megabytes. The translation quality has almost no loss and has already been applied to the real-time translation of live stream comments on Bilibili (B站), achieving a breakthrough from a lab demo to a high-concurrency real business scenario.
The base Hy-MT2-1.8B model originally belonged to the Tencent Hunyuan translation model family and supported bidirectional translation among 33 languages natively. In terms of overall translation quality, it has already exceeded several commercial translation APIs in the FLORES-200 general translation evaluation under the same size. However, due to the fact that the 1.8B model requires up to 3.3GB of memory in traditional FP16 precision, even when reduced to 4-bit quantization, it still needs over 1GB, which cannot meet the memory management requirements for multi-task concurrency on edge devices. To address this, Tencent has introduced two ultra-low bit quantization schemes.
The first scheme is a 2-bit model aimed at high-end devices. This approach uses Stretchable Elastic Quantization (SEQ) technology to quantize parameters to specific discrete values and combines Quantization-Aware Distillation (QAD) to achieve near-lossless translation quality while compressing the model size to 574MB.
The second scheme is an ultra-low bit 1.25-bit solution suitable for all device series. This scheme is based on Tencent's self-developed Sherry sparse-efficient ternary quantization technology, which was previously selected as an Oral paper at the top academic conference ACL2026. Its core strategy is to use a fine-grained sparsity mechanism, where three out of every four parameters are stored with specific values, and the rest are set to zero, resulting in an average of 1.25 bits per parameter. Combined with the STQ kernel designed specifically for CPUs, the original 3.3GB model was compressed to 440MB.
To free the ultra-low bit model from reliance on specific mobile ARM architectures, the Intel team conducted in-depth adaptation for the x86 ecosystem. By optimizing vectorization, weight reordering, and VNNI instruction fusion in quantized matrix operations, the low-bit quantization format of Hy-MT2 achieved significant improvements in token processing speed on Intel mainstream devices (including the latest third-generation Core Ultra and Core processors), successfully expanding the ultra-low bit solution from mobile devices to PCs and edge devices.
In practical business validation, Bilibili has officially integrated this ultra-low bit model into real-time translation of live stream comments. The total download size of the entire resource is approximately 600MB, and the memory usage during operation remains between 500 and 700MB. The average translation time for a single comment is around 500 to 800 milliseconds, which allows smooth real-time translation at regular comment frequencies and accurately handles various internet slang. This local independent operation mode on the device reduces cloud server call costs and bandwidth expenses, and avoids data uploading throughout the process, effectively protecting user privacy and security.
