Joint evaluation of sparse models and chips

In a separate public report, Moffett disclosed up to 2.89× execution speedup for a 7B model at 1:4 sparsity against the dense baseline in a same-platform compiler evaluation. Read the report.

The 1B result below comes from a different experiment. Under its specified conditions, the 1:4 configuration reaches approximately 2.67× compiler-estimated input throughput versus the dense configuration.

Model and evaluation conditions

At 1:4 sparsity, one weight in each group of four is retained in selected projection matrices. SparseMind contributes sparse training and model evaluation. Moffett provides the sparse processor, compiler and scheduling.

The 1B comparison uses 256 input tokens, BF16, four cores on one device, batch size one per core, 700 MHz and no CoMoLoRA branch. Compiler-estimated input throughput is 3,926, 6,613 and 10,483 tokens/s for dense, 2:4 and 1:4 respectively. Estimated execution latency is 97.678 ms for 1:4 and 260.796 ms for dense, a reduction of approximately 62.5%.

These input-stage figures are derived from compiler cycles and include the report’s IO/DMA scheduling. The 25% weight-retention ratio applies to selected projection layers. The full model also includes embeddings and an output head.

Model scales

With 256 input tokens, 1:4 configurations for 130M, 350M and 1B models achieve compiler-estimated speedups of 2.253×, 2.405× and 2.670× respectively. The 1B configuration with the auxiliary CoMoLoRA branch retains an estimated 2.40× speedup. Its quality impact needs evaluation under matching conditions.

Model quality

In a separate LLaMA-130M experiment, the 1:4 model with CoMoLoRA has mean validation perplexity of 25.0647, compared with 24.8140 for the dense reference. W8A16 simulated quantization changes the sparse model’s perplexity from 25.0647 to 25.0747. This quality experiment uses a different configuration from the 1B performance comparison.

Next steps

Further work will assess numerical consistency, device latency, energy use and production workloads, and explore larger models. Device measurements and deployment costs remain to be validated.