TL;DR

We’re excited to announce that AutoRound—Intel’s state‑of‑the‑art tuning‑based post‑training quantization (PTQ) algorithm—is now integrated into LLM Compressor. This collaboration delivers:

  • Higher accuracy for low bit-width quantization
  • Lightweight tuning (hundreds of steps, not thousands)
  • Zero additional inference overhead
  • Seamless compatibility with compressed-tensors and direct serving in vLLM

Broader quantization schemes and model coverage are coming next—try it now and help shape what we build.

What Is AutoRound?

AutoRound is an advanced post-training quantization (PTQ) algorithm designed for Large Language Models (LLMs) and Vision-Language Models (VLMs). It introduces three trainable parameters per quantized tensor: V (rounding offset/adjustment), α and β (learned clipping range controls). By processing decoder layers sequentially and applying signed gradient descent, AutoRound jointly optimizes rounding and clipping to minimize block‑wise output reconstruction error.

Core strengths:

  • Superior accuracy, especially at very low bit‑widths
  • Support multiple data types: W4A16, MXFP8, MXFP4, FP8, NVFP4, with more on the way
  • Mixed‑bit, layer‑wise precision search for flexible accuracy–efficiency trade‑offs
  • Applicability across both LLMs and VLMs

AutoRound enables quantized models in a range of low‑bit formats that are designed to accelerate inference on Intel® Xeon® processors, Intel® Gaudi® AI accelerators, Intel® Data Center GPUs, Intel® Arc™ B‑Series Graphics, as well as other GPUs (e.g., CUDA‑based devices).

Looking forward, Intel is adding native support for FP8, MXFP8, and MXFP4 formats to its next-generation Data Center GPUs, codenamed Crescent Island. Models quantized with AutoRound will naturally scale to take advantage of these data types across the Intel AI hardware portfolio. This creates a consistent path from algorithmic innovation to real‑world deployment.

For more details, please refer to the paper AutoRound (EMNLP 2024) and the GitHub repository intel/auto-round.

Why Integrate Into LLM Compressor?

LLM Compressor already provides a unified, modular system for compression primitives such as quantization and pruning. Integrating AutoRound into this ecosystem:

  • Aligns with the existing modifier architecture (e.g., GPTQModifier)
  • Reuses the sequential calibration and layer‑onloading infrastructure
  • Enables future interoperability with richer multi‑modifier recipes
  • Produces quantized models that are ready for vLLM serving, enabling a clean workflow from compression to deployment

Integration Overview

We completed the first stage of integration by introducing the new AutoRoundModifier into LLM Compressor, enabling production of wNa16 (e.g., W4A16) compressed models that seamlessly load in vLLM, as implemented in PR #1994. With a straightforward configuration—just specify your model and calibration data—you can quickly generate high‑quality low‑bit checkpoints. This initial stage supports quantizing a range of dense LLMs, including the Llama and Qwen model families, and demonstrates robust compatibility for practical deployment.

Try It Now (Quickstart)

1. Install

git clone https://github.com/vllm-project/llm-compressor.git
cd llm-compressor
pip install -e .

2. Load Model & Tokenizer

from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_ID = "Qwen/Qwen3-8B"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

3. Prepare Calibration Data

from auto_round.calib_dataset import get_dataset
NUM_CALIBRATION_SAMPLES = 128
MAX_SEQUENCE_LENGTH = 2048
ds = get_dataset(tokenizer=tokenizer,
                 seqlen=MAX_SEQUENCE_LENGTH,
                 nsamples=NUM_CALIBRATION_SAMPLES)

4. Run Quantization using AutoRound

The AutoRound quantization can run on a variety of devices, including CPUs and GPUs. Quantization and serving may not happen on the same device. For example, you can quantize on a workstation with GPU and later deploy on AIPC.

from llmcompressor import oneshot
from llmcompressor.modifiers.autoround import AutoRoundModifier

recipe = AutoRoundModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head"],
    iters=200,
)

oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    shuffle_calibration_samples=False,
)

SAVE_DIR = MODEL_ID.split("/")[-1] + "-W4A16-G128-AutoRound"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

In practice, 128 calibration samples + ~200 iterations often reach stable convergence. Increase the number of samples or iterations if you are targeting extremely low bits or tighter accuracy targets.

5. Serve in vLLM

Once quantization is complete, the same compressed model can be served on different hardware, independent of the device used for tuning. For example, you can serve the quantized Qwen3‑8B‑W4A16‑G128‑AutoRound model on a single Intel® Arc™ Pro B60 GPU:

vllm serve Qwen3-8B-W4A16-G128-AutoRound \
    --dtype=bfloat16 \
    --enforce-eager \
    --gpu-memory-util=0.8 \
    --no-enable-prefix-caching \
    --max-num-batched-tokens=8192

Note: please install vLLM from this PR https://github.com/vllm-project/vllm/pull/29484/

6. Evaluate (Example: GSM8K with lm_eval)

lm_eval --model vllm \
  --model_args pretrained="./Qwen3-8B-W4A16-G128-AutoRound,max_model_len=8192,max_num_batched_tokens=32768,max_num_seqs=128,gpu_memory_utilization=0.8,dtype=bfloat16,max_gen_toks=2048,enable_prefix_caching=False,enforce_eager=True" \
  --tasks gsm8k \
  --num_fewshot 5 \
  --limit 1000 \
  --batch_size 128

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.908|±  |0.0091|
|     |       |strict-match    |     5|exact_match|↑  |0.907|±  |0.0092|

Note: The results may fluctuate due to non-determinism.

Conclusion & Future Plans

With this first integration, AutoRound and LLM Compressor already provide a practical, production‑oriented path to low‑bit LLMs: W4A16 quantization is supported end‑to‑end, the workflow is simple to configure, and dense models such as Llama and Qwen are supported. The setup is robust, streamlined, and ready for practical deployment.

Looking ahead, we plan to extend support to additional schemes such as FP8, MXFP4, MXFP8, and NVFP4, add automatic mixed‑bit search for fine‑grained per‑layer optimization, and cover more model families, including Mixture‑of‑Experts (MoE) models. We also aim to deepen interoperability with other algorithms in LLM Compressor, which will allow AutoRound to combined into richer multi‑modifier recipes that serve both community use cases and Intel production workloads.

If you’d like to influence which formats, models, and workflows we prioritize next, please join the discussion in RFC #1968 and share your benchmarks or deployment requirements, or bring your feedback to the Intel Community so we can align the roadmap with real‑world needs.

Acknowledgements

We wish to acknowledge the contributions of the LLM Compressor community. Specifically, we thank Kyle Sayers, Dipika Sikka, Brian Dellabetta, Charles Hernandez, and Robert Shaw for their invaluable feedback on the early proposal and their diligent review of the pull requests.

RFC: https://github.com/vllm-project/llm-compressor/issues/1968

PRs:

  • https://github.com/vllm-project/llm-compressor/pull/1994
  • https://github.com/vllm-project/llm-compressor/pull/2055
  • https://github.com/vllm-project/llm-compressor/pull/2062
  • https://github.com/vllm-project/vllm/pull/29484/ (Under Review)