3.1-后训练量化(1)

3.1 后训练量化（PTQ）：GPTQ/AWQ/GPTQ-Intact

摘要

后训练量化（Post-Training Quantization, PTQ）是当前大模型压缩和优化的核心技术之一。本章将深入剖析GPTQ、AWQ、GPTQ-Intact等主流PTQ算法的原理、实现细节、性能对比，以及在各种硬件平台上的实际应用效果。这些技术能够在不重新训练模型的情况下，显著降低模型显存占用和推理延迟，为大规模部署提供重要支撑。

1. 量化基础理论

1.1 量化的数学基础

量化是将连续的浮点数映射到离散的整数表示过程。对于神经网络中的参数和激活值，量化可以表示为：

real_value = quantized_value × scale + zero_point
​

其中：

• quantized_value: 离散的整数值
• scale: 缩放因子
• zero_point: 零点偏移量

import torch
import numpy as np

class Quantization:
    def __init__(self, bits=8, symmetric=True):
        self.bits = bits
        self.symmetric = symmetric
        self.qmin = 0 if not symmetric else -(2**(bits-1))
        self.qmax = 2**bits - 1 if not symmetric else 2**(bits-1) - 1
    
    def quantize(self, tensor):
        """量化张量"""
        if self.symmetric:
            # 对称量化
            scale = torch.max(torch.abs(tensor)) / (2**(self.bits-1) - 1)
            q_tensor = torch.clamp(torch.round(tensor / scale), 
                                 self.qmin, self.qmax)
            return q_tensor, scale
        else:
            # 非对称量化
            min_val = torch.min(tensor)
            max_val = torch.max(tensor)
            scale = (max_val - min_val) / (self.qmax - self.qmin)
            zero_point = torch.round(-min_val / scale)
            q_tensor = torch.clamp(torch.round(tensor / scale) + zero_point, 
                                 self.qmin, self.qmax)
            return q_tensor, scale, zero_point
    
    def dequantize(self, q_tensor, scale, zero_point=None):
        """反量化"""
        if zero_point is not None:
            return q_tensor * scale - zero_point * scale
        else:
            return q_tensor * scale
​

1.2 误差分析与信息损失

量化误差主要包括：

1. 截断误差: 量化步长导致的精度损失
2. 舍入误差: 四舍五入操作
3. 溢出误差: 数值超出量化范围

def analyze_quantization_error(original, quantized, dequantized):
    """分析量化误差"""
    mse = torch.mean((original - dequantized) ** 2)
    mae = torch.mean(torch.abs(original - dequantized))
    max_error = torch.max(torch.abs(original - dequantized))
    
    # 计算信噪比
    signal_power = torch.mean(original ** 2)
    noise_power = mse
    snr = 10 * torch.log10(signal_power / noise_power) if noise_power > 0 else float('inf')
    
    # 计算相对误差
    relative_error = torch.mean(torch.abs((original - dequantized) / (original + 1e-8)))
    
    return {
        'mse': mse.item(),
        'mae': mae.item(),
        'max_error': max_error.item(),
        'snr': snr.item(),
        'relative_error': relative_error.item()
    }
​

1.3 量化精度对比

2. GPTQ算法详解

2.1 GPTQ核心原理

GPTQ（Group-wise Post-Training Quantization）是一种基于权重的逐组量化方法，其核心思想是：

1. 分组量化: 将权重矩阵按行分组
2. 顺序优化: 逐行量化，考虑后续行的依赖
3. 误差最小化: 量化误差的累计最小化

class GPTQ:
    def __init__(self, bits=4, perchannel=False, blocksize=128):
        self.bits = bits
        self.perchannel = perchannel
        self.blocksize = blocksize
        
    def quantize_layer(self, weight, activation_stats=None):
        """量化单个层"""
        if self.perchannel:
            # 按通道量化
            return self.perchannel_quantize(weight)
        else:
            # 按行分组量化
            return self.rowwise_quantize(weight)
    
    def rowwise_quantize(self, weight):
        """逐行量化"""
        out_features, in_features = weight.shape
        quantized_weight = torch.zeros_like(weight)
        
        # 计算激活统计量
        act_mean = torch.mean(weight, dim=1, keepdim=True)
        act_std = torch.std(weight, dim=1, keepdim=True)
        
        # 逐行量化
        for i in range(out_features):
            row = weight[i:i+1, :]
            
            # 计算当前行的量化参数
            if self.bits == 8:
                scale = act_std[i] / 127.0
                zero_point = 0  # 对称量化
            elif self.bits == 4:
                scale = act_std[i] / 7.0
                zero_point = 0
            
            # 量化当前行
            q_row = torch.clamp(torch.round(row / scale), -8, 7) if self.bits == 4 else \
                    torch.clamp(torch.round(row / scale), -127, 127)
            
            # 保存量化参数
            quantized_weight[i:i+1, :] = q_row
            
            # 更新激活统计量（考虑量化影响）
            if i < out_features - 1:
                next_row = weight[i+1:i+2, :]
                next_row -= (next_row @ q_row.T) @ q_row / (q_row @ q_row.T + 1e-6)
        
        return quantized_weight
    
    def perchannel_quantize(self, weight):
        """按通道量化"""
        if weight.dim() == 2:
            # 全连接层
            out_features, in_features = weight.shape
            quantized_weight = torch.zeros_like(weight)
            
            for i in range(out_features):
                row = weight[i:i+1, :]
                
                # 计算该行的量化参数
                row_max = torch.max(torch.abs(row))
                scale = row_max / (2**(self.bits-1) - 1)
                
                # 量化
                q_row = torch.clamp(torch.round(row / scale), 
                                 -(2**(self.bits-1)-1), 2**(self.bits-1)-1)
                
                quantized_weight[i:i+1, :] = q_row * scale
            
            return quantized_weight
​

2.2 GPTQ的数学推导

GPTQ的目标是最小化量化误差：

minimize Σ||W - W_q||_F^2
​

其中W_q是量化后的权重矩阵。通过顺序优化，每行量化时考虑后续行的依赖：

W_q[i] = round((W[i] - Σ_{j>i} W[j] @ W_q[j]^T @ W_q[j] / (W_q[j] @ W_q[j]^T)) / σ_i)
​

def gptq_optimization_step(weight, quantized_weight, row_idx, scale):
    """GPTQ优化步骤"""
    # 获取当前行和已量化行
    current_row = weight[row_idx:row_idx+1, :]
    quantized_rows = quantized_weight[row_idx+1:, :]
    
    # 计算累积误差
    if quantized_rows.size(0) > 0:
        # 计算后续量化行对当前行的影响
        error = torch.sum(quantized_rows.T @ current_row - 
                        quantized_rows.T @ quantized_rows, dim=1, keepdim=True)
        
        # 减去累积误差
        corrected_row = current_row - error / torch.sum(quantized_rows ** 2, dim=1, keepdim=True)
    else:
        corrected_row = current_row
    
    # 量化
    quantized_row = torch.clamp(torch.round(corrected_row / scale), 
                              -8, 7)  # 假设4-bit量化
    
    return quantized_row * scale
​

2.3 GPTQ实现细节

class FullGPTQ:
    def __init__(self, model, quant_config=None):
        self.model = model
        self.quant_config = quant_config or {
            'bits': 4,
            'perchannel': True,
            'blocksize': 128,
            'damp_percent': 0.1,
            'act_order': True
        }
        
    def quantize_model(self, calibration_data):
        """量化整个模型"""
        # 收集所有权重
        weights = {}
        for name, module in self.model.named_modules():
            if hasattr(module, 'weight'):
                weights[name] = module.weight.data.clone()
        
        # 逐层量化
        quantized_weights = {}
        for name, weight in weights.items():
            print(f"Quantizing layer: {name}")
            quantized_weight = self.quantize_layer(weight, calibration_data)
            quantized_weights[name] = quantized_weight
        
        # 替换模型权重
        for name, module in self.model.named_modules():
            if hasattr(module, 'weight') and name in quantized_weights:
                module.weight.data = quantized_weights[name]
        
        return quantized_weights
    
    def quantize_layer(self, weight, calibration_data):
        """量化单层权重"""
        # 获取层信息
        layer_name = self._get_layer_name(weight)
        layer_type = self._get_layer_type(layer_name)
        
        if layer_type == 'linear':
            return self.quantize_linear(weight, calibration_data)
        elif layer_type == 'attention':
            return self.quantize_attention(weight, calibration_data)
        else:
            return weight  # 暂不量化其他类型
    
    def quantize_linear(self, weight, calibration_data):
        """量化线性层"""
        # 计算激活统计量
        if calibration_data is not None:
            act_stats = self._compute_activation_stats(calibration_data)
        else:
            act_stats = {'mean': 0, 'std': 1}
        
        # 选择量化策略
        if self.quant_config['perchannel']:
            return self._perchannel_quantize(weight, act_stats)
        else:
            return self._groupwise_quantize(weight, act_stats)
    
    def _perchannel_quantize(self, weight, act_stats):
        """按通道量化"""
        out_features, in_features = weight.shape
        quantized = torch.zeros_like(weight)
        
        # 按行量化
        for i in range(out_features):
            row = weight[i:i+1, :]
            
            # 考虑激活统计
            if self.quant_config['act_order']:
                scale = act_stats['std'][i] / (2**(self.quant_config['bits']-1) - 1)
            else:
                scale = torch.max(torch.abs(row)) / (2**(self.quant_config['bits']-1) - 1)
            
            # 量化
            q_row = torch.clamp(torch.round(row / scale), 
                              -(2**(self.quant_config['bits']-1)-1), 
                              2**(self.quant_config['bits']-1)-1)
            
            quantized[i:i+1, :] = q_row * scale
        
        return quantized
​

3. AWQ算法详解

3.1 AWQ核心原理

AWQ（Activation-aware Weight Quantization）是一种激活感知的权重量化方法，其核心思想是：

1. 激活重要性分析: 分析激活值对输出的重要性
2. 重要性感知量化: 对重要性高的部分保持更高精度
3. 动态量化: 根据激活分布调整量化参数

class AWQ:
    def __init__(self, bits=4, importance_threshold=0.1):
        self.bits = bits
        self.importance_threshold = importance_threshold
        self.importance_map = {}
        
    def compute_activation_importance(self, activations, gradients):
        """计算激活重要性"""
        # 基于梯度和激活值计算重要性
        importance = torch.abs(activations * gradients)
        
        # 归一化
        importance = importance / (torch.max(importance) + 1e-8)
        
        return importance
    
    def quantize_with_importance(self, weight, activations):
        """基于重要性的量化"""
        # 计算权重重要性
        weight_importance = torch.abs(weight)
        
        # 计算激活重要性
        act_importance = self._compute_activation_importance(activations)
        
        # 综合重要性
        combined_importance = weight_importance * act_importance
        
        # 动态调整量化精度
        importance_mask = combined_importance > self.importance_threshold
        
        # 量化
        quantized_weight = self._adaptive_quantize(weight, importance_mask)
        
        return quantized_weight, importance_mask
    
    def _adaptive_quantize(self, weight, importance_mask):
        """自适应量化"""
        # 高重要性部分使用更高精度
        high_importance = weight * importance_mask
        low_importance = weight * (~importance_mask)
        
        # 高精度量化（8-bit）
        if self.bits == 4:
            high_q = self._quantize(high_importance, 8)
            low_q = self._quantize(low_importance, 4)
        
        quantized = high_q + low_q
        
        return quantized
    
    def _quantize(self, tensor, target_bits):
        """量化到指定精度"""
        scale = torch.max(torch.abs(tensor)) / (2**(target_bits-1) - 1)
        q_tensor = torch.clamp(torch.round(tensor / scale), 
                             -(2**(target_bits-1)-1), 2**(target_bits-1)-1)
        return q_tensor * scale
​

3.2 AWQ的数学优化目标

AWQ的优化目标是最小化带权重的量化误差：

minimize Σ_i w_i ||W[i] - W_q[i]||^2
​

其中w_i是权重i的重要性权重。

def awq_objective_function(weight, quantized_weight, importance_weights):
    """AWQ目标函数"""
    error = torch.sum(importance_weights * (weight - quantized_weight) ** 2)
    return error

def optimize_quantization_with_importance(weight, importance_weights, bits=4):
    """基于重要性的量化优化"""
    # 初始量化
    scale = torch.max(torch.abs(weight)) / (2**(bits-1) - 1)
    initial_quantized = torch.clamp(torch.round(weight / scale), 
                                  -(2**(bits-1)-1), 2**(bits-1)-1) * scale
    
    # 优化迭代
    optimized_quantized = initial_quantized.clone()
    learning_rate = 0.01
    
    for _ in range(100):  # 迭代优化
        # 计算梯度
        error = awq_objective_function(weight, optimized_quantized, importance_weights)
        gradient = -2 * importance_weights * (weight - optimized_quantized)
        
        # 更新量化值
        optimized_quantized -= learning_rate * gradient
        
        # 重新量化
        optimized_quantized = torch.clamp(optimized_quantized, 
                                        -(2**(bits-1)-1) * scale, 
                                        (2**(bits-1)-1) * scale)
    
    return optimized_quantized
​

3.3 AWQ实现细节

class FullAWQ:
    def __init__(self, model, quant_config=None):
        self.model = model
        self.quant_config = quant_config or {
            'bits': 4,
            'importance_threshold': 0.1,
            'alpha': 0.5,
            'beta': 0.5
        }
        
    def calibrate_importance(self, calibration_dataset):
        """校准重要性权重"""
        importance_scores = {}
        
        # 前向传播收集激活
        activations = self._collect_activations(calibration_dataset)
        
        # 计算重要性
        for layer_name, layer_activations in activations.items():
            # 反向传播获取梯度
            gradients = self._compute_gradients(layer_activations)
            
            # 计算重要性
​