Demystifying LLM Model Formats: GGUF, GPTQ, AWQ, EXL2, and Beyond Explained

The Crucial Role of LLM Model Formats and Quantization

At a glance, In the rapidly evolving world of Large Language Models (LLMs), deploying and running these powerful AI tools efficiently is paramount. Whether you’re a developer, researcher, or enthusiast, understanding the various model formats and quantization techniques available can dramatically impact performance, memory footprint, and even the hardware you need. This guide will demystify the leading LLM model formats like GGUF, GPTQ, AWQ, EXL2, and more, helping you make informed decisions for your projects.

Containers vs. Quantization Methods: The Core Distinction

A common source of confusion arises from conflating two distinct concepts:

• Containers: These define how the model’s tensors (data structures) are organized and stored on disk. Think of them as the packaging. Examples include Safetensors and GGUF.
• Quantization Methods: These are techniques that compress the model’s weights into fewer bits, significantly reducing file size and memory usage, often with minimal impact on accuracy. Examples include GPTQ, AWQ, and bitsandbytes NF4.

Sometimes, a format might combine both a quantization method and a specific storage layout, like EXL2 and EXL3, which are tightly coupled to their respective inference libraries.

Understanding Memory Footprint: A Quick Rule of Thumb

To estimate the memory required for a model’s weights, use this simple formula:

Weight memory ≈ parameters × bits-per-weight ÷ 8

For example, For instance, a 70B parameter model at full 16-bit precision requires around 140 GB, while a highly quantized version (e.g., ~4.5 bits per weight) could drop to approximately 39 GB. Remember, this calculation only covers weights; the KV cache and runtime overhead will add to the total memory consumption.

Key LLM Model Formats and Quantization Techniques

1. Full Precision: Safetensors and PyTorch .bin

Unquantized models typically ship with 16-bit weights, stored either as pytorch_model.bin or model.safetensors.

• PyTorch .bin / .pt files: These older formats use Python pickle, which poses a security risk as loading them can execute arbitrary code from untrusted sources.
• Safetensors: Developed by Hugging Face, Safetensors mitigates this risk. It’s a secure, non-executable binary format with a JSON header and raw tensor buffers. Tensors can be memory-mapped and loaded individually, enhancing efficiency and security. Safetensors is now a PyTorch Foundation project.

That said, Notably most quantized models (GPTQ, AWQ, EXL2, EXL3, MLX) often utilize .safetensors as their underlying container. The quantization itself resides within the tensor data and a configuration file, not in a new container format.

2. GGUF (GGML Unified Format)

GGUF is a binary format primarily used for running models with GGML and its executors, most notably llama.cpp. Created by Georgi Gerganov (who also leads llama.cpp), it was introduced on August 21, 2023, as the successor to the older GGML format.

• Why it replaced GGML: The previous GGML, GGMF, and GGJT formats lacked architectural metadata, making them prone to breakage with new hyperparameters. GGUF adopted typed key-value metadata, allowing for future extensibility without invalidating older files.
• Design Goals: GGUF aims for single-file deployment, extensibility, memory-map (mmap) compatibility, easy loading, and self-contained information. Unlike tensor-only formats, GGUF can store the tokenizer, special tokens, and chat templates alongside the weights.
• Reading GGUF Quant Names: Suffixes like Q4_K_M.gguf indicate the quantization scheme. These names often denote mixed quantization levels, with _S, _M, and _L referring to different mixes. For instance, Q4_K_M might use Q6_K for specific critical layers and Q4_K elsewhere. Q8_0, despite being labeled ‘legacy’ by Hugging Face, remains a widely used near-lossless GGUF option.
• Quality vs. Size: GGUF offers various quantization levels, allowing users to balance model size and inference quality. Lower bitrates (e.g., Q4_K_M) significantly reduce size but incur a higher perplexity change compared to full precision or Q8_0.
• Importance Matrix (imatrix): GGUF supports calibration data through an importance matrix. Tools like llama-imatrix can compute this from a text file, which llama-quantize then uses to improve the quality of lower-bit quantizations.
• Where GGUF Runs: GGUF is native to the llama.cpp ecosystem, including LM Studio, GPT4All, and Ollama. While vLLM has experimental and limited GGUF support, it often requires an out-of-tree plugin.

3. GPTQ: Post-Training Quantization

Interestingly, GPTQ, developed by Elias Frantar et al., debuted on arXiv in October 2022 and was published at ICLR 2023. It’s a highly effective one-shot, post-training weight quantization method.

• How it Works: GPTQ uses approximate second-order (Hessian) information to intelligently round weights. It compensates for rounding errors by adjusting unquantized weights, requiring a small calibration dataset but no retraining.
• Main Results: GPTQ can quantize large 175B parameter models down to 3 or 4 bits per weight with negligible accuracy loss, achieving significant end-to-end speedups (e.g., 3.25x on NVIDIA A100 over FP16).
• Reading GPTQ Names: Repositories often include tags like GPTQ or 4bit-128g. The 128g denotes a group size of 128 weights per scale, where smaller groups generally improve accuracy at the cost of slight size increase. desc_act indicates activation-order quantization, which quantizes columns by importance for better accuracy.
• Tooling Status in 2026: The original AutoGPTQ library is no longer maintained. GPTQModel has emerged as the recommended successor, fully integrating with Transformers, Optimum, and PEFT, and its output runs across Transformers, vLLM, and SGLang. Calibration for an 8B model can take around 20 minutes on an A100 GPU.

4. AWQ: Activation-aware Weight Quantization

AWQ, from Song Han’s group at MIT, appeared on arXiv in June 2023 and earned the MLSys 2024 Best Paper Award. Its core innovation lies in identifying and protecting critical weights.

• Core Idea: AWQ posits that not all weights contribute equally to model performance. By analyzing activation magnitudes, it identifies a small percentage (around 1%) of ‘salient’ weights. Instead of storing these at higher precision, AWQ scales them up through a mathematically equivalent transformation, maintaining a uniform, hardware-friendly format. This method avoids backpropagation or reconstruction, reducing the risk of overfitting to the calibration set.
• Speed and Cost: AWQ boasts impressive speedups, with TinyChat runtime showing more than 3x faster performance than Hugging Face’s FP16 on various GPUs. Its calibration process is also notably faster than GPTQ, taking approximately 10 minutes for an 8B model on an A100.
• Tooling Status in 2026: Similar to GPTQ, AutoAWQ is now officially deprecated. The recommended workflow for AWQ is through llm-compressor, integrated into vLLM. MLX-LM also supports AWQ on Apple Silicon.

5. EXL2 (ExLlamaV2): Fine-Grained Quantization for Consumer GPUs

However, EXL2 is the native format for ExLlamaV2, an inference library developed by turboderp specifically for consumer-grade NVIDIA GPUs. It supports a wide range of quantization from 2 to 8 bits.

• What Makes it Different: EXL2 offers unparalleled flexibility in bitrates. It allows for any average bitrate between 2 and 8 bits per weight, with quantization levels mixed across and even within layers. More important columns within a layer can receive more bits. Its converter automatically allocates bits by measuring error against calibration data, minimizing worst-case error while hitting a target bitrate. This is why EXL2 files carry precise names like 4.65bpw.
• Runtime: TabbyAPI is the official server, offering an OpenAI-compatible API. EXL2 renames some tensors internally to appear as Llama variants, which can make it challenging to reuse in other frameworks.

6. EXL3 (ExLlamaV3): The Next Evolution

EXL3 is the successor to EXL2, built upon the QTIP (Trellis-Coded Quantization with Incoherence Processing) method from Cornell RelaxML, published at NeurIPS 2024. EXL3 retains QTIP’s procedural codebook and trellis encoding while refining tensor regularization and packing.

• Why it Matters: EXL3 offers a streamlined conversion process – simply supply a Hugging Face model and a target bitrate, with Hessians computed on the fly. Conversion is reasonably fast, taking minutes for smaller models and a few hours for 70B+ models on a single consumer GPU (e.g., RTX 4090). It achieves remarkably low bitrates; a Llama-3.1-70B can remain coherent at 1.6 bits per weight, fitting into under 16 GB of VRAM with a 3-bit output layer and 4,096-token cache. Unlike EXL2, EXL3 largely preserves the original tensor structure, making it more portable.
• Features: ExLlamaV3 introduces 2-8 bit KV-cache quantization, tensor and expert-parallel inference, speculative decoding, multimodal support, and a Transformers plugin. Recent updates include CPU offloading for large MoE models.
• Hardware Note: ExLlamaV3 requires CUDA 12.4 or later.

7. bitsandbytes (NF4 / INT8): On-the-Fly Quantization

Meanwhile, bitsandbytes is unique in that it’s typically used for on-the-fly quantization rather than pre-quantized downloads. You load a 16-bit model and quantize it dynamically.

• QLoRA Connection: Its 4-bit mode originates from QLoRA, utilizing NF4 (4-bit NormalFloat) – a data type optimized for normally distributed weights. It also employs double quantization, where quantization constants themselves are quantized, further saving memory. This approach enables fine-tuning a 65B model on a single 48 GB GPU with performance matching 16-bit fine-tuning.
• Hugging Face Guidance: bitsandbytes requires no calibration dataset. While it enables memory savings, inference speedup is not guaranteed. It remains the standard pathway for QLoRA fine-tuning via PEFT.

8. MLX (Apple Silicon)

MLX-LM is a Python package from Apple Machine Learning Research designed for running and fine-tuning LLMs specifically on Apple Silicon hardware. MLX models are essentially Safetensors files containing MLX-specific quantized weights. The mlx_lm.convert tool can quantize Hugging Face models and upload them to the mlx-community organization.

In practical terms, For Mac users, both GGUF (via llama.cpp) and MLX represent robust options for local LLM inference.

9. Other Notable Formats and Methods

• compressed-tensors / FP8: This is the on-disk format generated by llm-compressor, supporting FP8, INT4/INT8 weight-only schemes, NVFP4, and sparsity. FP8 requires newer hardware like NVIDIA H100/H200/B100 or AMD MI300 to fully realize its benefits.
• HQQ: A fast, calibration-free quantization method that can go from 8 down to 1 bit, though accuracy can sharply decline below 4 bits.
• SINQ: Another calibration-free, on-the-fly method now integrated into Transformers.
• AQLM, SpQR, VPTQ, HIGGS: These are advanced research methods pushing the boundaries of extremely low bitrates, often below 2 bits per weight.

Comparison at a Glance

• Safetensors (16-bit)
  • What it is: Secure container for full-precision models.
  • Calibration: None.
  • Best Hardware: GPUs with sufficient VRAM.
  • Main Runtimes: Transformers, vLLM, SGLang.
• GGUF
  • What it is: Container + various quantization types for GGML ecosystem.
  • Calibration: Optional (imatrix).
  • Best Hardware: CPU, Apple Silicon, CPU+GPU split.
  • Main Runtimes: llama.cpp, Ollama, LM Studio.
• GPTQ
  • What it is: Post-training weight quantization method (in safetensors).
  • Calibration: Required.
  • Best Hardware: GPUs.
  • Main Runtimes: vLLM, SGLang, Transformers (via GPTQModel).
• AWQ
  • What it is: Activation-aware weight quantization method (in safetensors).
  • Calibration: Required.
  • Best Hardware: GPUs.
  • Main Runtimes: vLLM, SGLang, Transformers (via llm-compressor).
• EXL2
  • What it is: Quantization method + custom storage layout for ExLlamaV2.
  • Calibration: Required.
  • Best Hardware: Consumer NVIDIA GPUs.
  • Main Runtimes: ExLlamaV2, TabbyAPI.
• EXL3
  • What it is: Advanced quantization method + custom storage layout for ExLlamaV3.
  • Calibration: Built into conversion.
  • Best Hardware: Consumer NVIDIA GPUs (CUDA 12.4+).
  • Main Runtimes: ExLlamaV3, TabbyAPI.
• bitsandbytes NF4
  • What it is: On-the-fly quantization method (for QLoRA).
  • Calibration: None.
  • Best Hardware: NVIDIA (and Intel) GPUs.
  • Main Runtimes: Transformers, PEFT.
• MLX
  • What it is: Quantization method (in safetensors) for Apple Silicon.
  • Calibration: None by default.
  • Best Hardware: Apple Silicon.
  • Main Runtimes: MLX-LM.

Choosing the Right Format for Your Needs

Selecting the optimal LLM format depends on your specific hardware, performance goals, and use case:

• For Mac, CPU-only, or models larger than your VRAM: GGUF is your go-to. Start with Q4_K_M and scale up to Q5_K_M or Q6_K if memory permits for better quality.
• For serving multiple users on data-center GPUs: Consider AWQ or GPTQ within vLLM/SGLang for efficient inference. For cutting-edge performance, FP8 on Hopper or Blackwell-class cards is ideal.
• For single-user inference on consumer NVIDIA GPUs, prioritizing maximum tokens per second: EXL3 via TabbyAPI is the top choice (EXL2 for older setups).
• For fine-tuning on a budget: bitsandbytes NF4 with QLoRA provides an excellent balance of performance and memory efficiency.
• For Apple Silicon users seeking a Python-native workflow or fine-tuning: MLX offers a streamlined experience.

Expert Perspective

From an industry angle, the clearest signal around LLM model formats is how it may influence gguf. The story reads less like a one-day spike and more like a marker of broader movement.

The next phase will depend on how quickly teams, regulators, or customers react. In practice, that gives LLM model formats room to reshape expectations across model over the near term.

For readers focused on practical impact, the best next step is to watch what changes around quantization once attention turns into execution.

Frequently Asked Questions

Why does LLM model formats matter right now?

The Crucial Role of LLM Model Formats and QuantizationAt a glance, In the rapidly evolving world of Large Language Models (LLMs), deploying and running these powerful AI tools efficiently is paramount.

What broader change could LLM model formats signal?

Whether you’re a developer, researcher, or enthusiast, understanding the various model formats and quantization techniques available can dramatically impact performance, memory footprint, and even the hardware you need.

What should the market watch next around LLM model formats?

This guide will demystify the leading LLM model formats like GGUF, GPTQ, AWQ, EXL2, and more, helping you make informed decisions for your projects.Containers vs.

Key Takeaways

• Always distinguish between a file format (like GGUF or Safetensors) and a quantization method (like GPTQ or AWQ).
• GGUF is the standard for local inference on CPUs, Apple Silicon, and mixed CPU+GPU setups.
• GPTQ and AWQ are the primary 4-bit workhorses for GPU serving in frameworks like vLLM, SGLang, and Transformers.
• EXL2 and especially EXL3 are engineered for blazing-fast single-user inference and highly granular bitrates on consumer NVIDIA GPUs.
• Be aware that older libraries like AutoGPTQ and AutoAWQ are deprecated; opt for their modern replacements like GPTQModel or llm-compressor.

Source: https://www.marktechpost.com/2026/09/18/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026/