This is the INT4 Llama-3-8b model quantized by per-channel QQQ. QQQ is an innovative and hardware-optimized W4A8 quantization solution. For more details, please refer to our code repo and our paper.

Downloads last month: 7

Safetensors

Model size

2B params

Tensor type

F16

F32

I32

Paper for HandH1998/QQQ-Llama-3-8b

QQQ: Quality Quattuor-Bit Quantization for Large Language Models

Paper • 2406.09904 • Published Jun 14, 2024 • 1