What is LLM Quantization?#

Quantization formats (like GGUF, AWQ, and EXL2) compress floating-point weights into lower-bit representations, allowing large models to run smoothly on consumer hardware and local GPUs.

Everyday Analogy#

Converting an uncompressed high-resolution bitmap image into an optimized, visually indistinguishable JPEG to save disk space.

Practical Applications#

  • Running open-source 70B parameter models on personal workstations
  • Edge device inference on smartphones and embedded hardware