Joint evaluation of sparse models and chips

In a separate public report, Moffett disclosed up to 2.89× execution speedup for a 7B model at 1:4 sparsity against the dense baseline in a same-platform compiler evaluation. Read the report.

The 1B result below comes from a different experiment. Under its specified conditions, the 1:4 configuration reaches approximately 2.67× compiler-estimated input throughput versus the dense configuration.

Model and evaluation conditions

At 1:4 sparsity, one weight in each group of four is retained in selected projection matrices. The joint evaluation connects sparse models with the target platform’s compiled execution path.

The 1B comparison uses 256 input tokens, BF16, four cores on one device, batch size one per core, 700 MHz and no CoMoLoRA branch. Compiler-estimated input throughput is 3,926, 6,613 and 10,483 tokens/s for dense, 2:4 and 1:4 respectively. Estimated execution latency is 97.678 ms for 1:4 and 260.796 ms for dense, a reduction of approximately 62.5%.

These input-stage figures are derived from compiler cycles and include the report’s IO/DMA scheduling. The 25% weight-retention ratio applies to selected projection layers. The full model also includes embeddings and an output head.

Model scales

With 256 input tokens, 1:4 configurations for 130M, 350M and 1B models achieve compiler-estimated speedups of 2.253×, 2.405× and 2.670× respectively. The 1B configuration with the auxiliary CoMoLoRA branch retains an estimated 2.40× speedup. Its quality impact needs evaluation under matching conditions.

Model quality

In a separate LLaMA-130M experiment, the 1:4 model with CoMoLoRA has mean validation perplexity of 25.0647, compared with 24.8140 for the dense reference. W8A16 simulated quantization changes the sparse model’s perplexity from 25.0647 to 25.0747. This quality experiment uses a different configuration from the 1B performance comparison.

Next steps

These are compiler-level results. Device latency, energy use and deployment benefits under production workloads remain to be validated.