ARGaze：自回归Transformer实现在线注视点估计

arXiv 2602.05132 | 2026年2月

一、研究背景

1.1 注视点估计的重要性

注视点（Gaze）估计是驾驶员监控系统的核心技术：

应用场景：

• 视线偏离道路检测（Euro NCAP DSM核心指标）
• 认知分心判断（视线停留在后视镜但未移动）
• HMI免触交互（眼动选择菜单）
• 警告优先级判断（视线是否在碰撞风险方向）

1.2 传统方法局限

方法	问题
基于几何模型	需要精确标定，对头位敏感
基于CNN回归	单帧预测，缺乏时序一致性
视频CNN（3D）	计算重，延迟高

核心挑战：

• 注视点具有强时序依赖性（眼睛不可能瞬间跳跃）
• 实时推理需要低延迟（<10ms）
• 跨个体泛化困难

1.3 ARGaze创新点

将注视点估计重新定义为序列预测问题：

flowchart LR
    subgraph 传统方法
        A1[单帧图像] --> B1[CNN回归]
        B1 --> C1[注视点坐标]
    end
    
    subgraph ARGaze
        A2[当前帧图像] --> B2[视觉特征提取]
        D[历史注视序列] --> E[Transformer解码器]
        B2 --> E
        E --> F[当前注视点]
        
        F --> G[滑动窗口缓存]
        G --> D
    end
    
    style ARGaze fill:#e8f5e9

二、模型架构

2.1 自回归注视点预测

核心思想：利用历史注视点序列预测当前位置，类似语言模型预测下一个词。

"""
ARGaze: 自回归注视点估计模型
"""

import torch
import torch.nn as nn
import torch.nn.functional as F
import math


class ARGaze(nn.Module):
    """
    ARGaze: Autoregressive Gaze Estimation
    
    将注视点估计建模为序列预测问题
    """
    
    def __init__(self, config):
        """
        Args:
            config: 模型配置
        """
        super().__init__()
        
        self.config = config
        
        # 视觉编码器（提取图像特征）
        self.visual_encoder = VisualEncoder(
            img_size=config['img_size'],
            patch_size=config['patch_size'],
            d_model=config['d_model']
        )
        
        # 注视点嵌入（将坐标转换为向量）
        self.gaze_embed = nn.Linear(2, config['d_model'])  # (x, y) -> d_model
        
        # 位置编码
        self.pos_encoding = PositionalEncoding(config['d_model'])
        
        # Transformer解码器
        decoder_layer = nn.TransformerDecoderLayer(
            d_model=config['d_model'],
            nhead=config['nhead'],
            dim_feedforward=config['dim_feedforward'],
            dropout=config['dropout'],
            batch_first=True
        )
        self.transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=config['num_layers'])
        
        # 输出投影（预测注视点坐标）
        self.gaze_head = nn.Linear(config['d_model'], 2)  # d_model -> (x, y)
        
        # 滑动窗口缓存
        self.history_len = config['history_len']
        self.register_buffer('gaze_history', None)
        
    def forward(self, image, gaze_history=None):
        """
        前向传播
        
        Args:
            image: [B, C, H, W] 当前帧
            gaze_history: [B, T, 2] 历史注视点（可选）
        
        Returns:
            gaze_pred: [B, 2] 预测的注视点
        """
        B = image.size(0)
        
        # 1. 提取视觉特征
        visual_feat = self.visual_encoder(image)  # [B, num_patches, d_model]
        
        # 2. 准备历史注视点
        if gaze_history is None:
            gaze_history = torch.zeros(B, self.history_len, 2, device=image.device)
        
        # 3. 注视点嵌入
        gaze_embed = self.gaze_embed(gaze_history)  # [B, T, d_model]
        gaze_embed = self.pos_encoding(gaze_embed)
        
        # 4. Transformer解码（以历史注视点为查询，视觉特征为记忆）
        decoded = self.transformer_decoder(
            tgt=gaze_embed,      # 历史注视点
            memory=visual_feat   # 视觉特征
        )  # [B, T, d_model]
        
        # 5. 取最后时刻输出预测
        last_output = decoded[:, -1, :]  # [B, d_model]
        
        # 6. 预测注视点
        gaze_pred = self.gaze_head(last_output)  # [B, 2]
        
        return gaze_pred
    
    def init_history(self, batch_size, device):
        """初始化历史缓存"""
        self.gaze_history = torch.zeros(batch_size, self.history_len, 2, device=device)
    
    def update_history(self, new_gaze):
        """更新历史缓存"""
        if self.gaze_history is None:
            return
        
        # 滑动窗口：移除最早，添加最新
        self.gaze_history = torch.cat([
            self.gaze_history[:, 1:, :],
            new_gaze.unsqueeze(1)
        ], dim=1)


class VisualEncoder(nn.Module):
    """视觉编码器（ViT风格）"""
    
    def __init__(self, img_size=224, patch_size=16, d_model=256):
        super().__init__()
        
        self.patch_size = patch_size
        self.num_patches = (img_size // patch_size) ** 2
        
        # Patch嵌入
        self.patch_embed = nn.Conv2d(3, d_model, kernel_size=patch_size, stride=patch_size)
        
        # 位置编码
        self.pos_embed = nn.Parameter(torch.zeros(1, self.num_patches, d_model))
        nn.init.trunc_normal_(self.pos_embed, std=0.02)
        
        # Transformer编码器
        encoder_layer = nn.TransformerEncoderLayer(d_model=d_model, nhead=8, batch_first=True)
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=4)
        
    def forward(self, x):
        """
        Args:
            x: [B, C, H, W]
        Returns:
            feat: [B, num_patches, d_model]
        """
        # Patch嵌入
        x = self.patch_embed(x)  # [B, d_model, H/patch, W/patch]
        x = x.flatten(2).transpose(1, 2)  # [B, num_patches, d_model]
        
        # 添加位置编码
        x = x + self.pos_embed
        
        # Transformer编码
        x = self.transformer(x)
        
        return x


class PositionalEncoding(nn.Module):
    """正弦位置编码"""
    
    def __init__(self, d_model, max_len=500):
        super().__init__()
        
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        
        self.register_buffer('pe', pe.unsqueeze(0))
        
    def forward(self, x):
        """Args: x [B, T, d_model]"""
        return x + self.pe[:, :x.size(1)]


# 模型测试
if __name__ == "__main__":
    config = {
        'img_size': 224,
        'patch_size': 16,
        'd_model': 256,
        'nhead': 8,
        'dim_feedforward': 512,
        'dropout': 0.1,
        'num_layers': 4,
        'history_len': 16
    }
    
    model = ARGaze(config)
    
    # 模拟输入
    B = 4
    image = torch.randn(B, 3, 224, 224)
    gaze_history = torch.randn(B, config['history_len'], 2)
    
    # 预测
    gaze_pred = model(image, gaze_history)
    
    print(f"Image shape: {image.shape}")
    print(f"Gaze history shape: {gaze_history.shape}")
    print(f"Predicted gaze shape: {gaze_pred.shape}")
    print(f"Model parameters: {sum(p.numel() for p in model.parameters()):,}")

2.2 自回归训练策略

"""
自回归训练：教师强制与计划采样
"""

import torch
import torch.nn as nn
import numpy as np


class ARGazeTrainer:
    """ARGaze训练器"""
    
    def __init__(self, model, config):
        self.model = model
        self.config = config
        
        # 损失函数
        self.criterion = nn.MSELoss()
        
        # 优化器
        self.optimizer = torch.optim.AdamW(model.parameters(), lr=config['lr'])
        
        # 计划采样参数
        self.teacher_forcing_ratio = 1.0  # 初始完全教师强制
        
    def train_epoch(self, dataloader, epoch):
        """训练一个epoch"""
        
        self.model.train()
        total_loss = 0
        
        # 更新教师强制比例
        self.teacher_forcing_ratio = max(0.0, 1.0 - epoch * 0.1)
        
        for batch in dataloader:
            images, gaze_sequences = batch
            # images: [B, T, C, H, W]
            # gaze_sequences: [B, T, 2]
            
            B, T, _, _, _ = images.shape
            
            batch_loss = 0
            
            # 初始化历史
            gaze_history = torch.zeros(B, self.config['history_len'], 2, device=images.device)
            
            for t in range(T):
                # 当前帧
                current_image = images[:, t]
                
                # 预测
                gaze_pred = self.model(current_image, gaze_history)
                
                # 真实值
                gaze_true = gaze_sequences[:, t]
                
                # 损失
                loss = self.criterion(gaze_pred, gaze_true)
                batch_loss += loss
                
                # 更新历史（教师强制 vs 预测值）
                if np.random.random() < self.teacher_forcing_ratio:
                    # 教师强制：使用真实值
                    gaze_history = self._update_history(gaze_history, gaze_true)
                else:
                    # 自回归：使用预测值
                    gaze_history = self._update_history(gaze_history, gaze_pred.detach())
            
            # 反向传播
            self.optimizer.zero_grad()
            batch_loss.backward()
            self.optimizer.step()
            
            total_loss += batch_loss.item()
        
        return total_loss / len(dataloader)
    
    def _update_history(self, history, new_gaze):
        """更新历史窗口"""
        return torch.cat([
            history[:, 1:, :],
            new_gaze.unsqueeze(1)
        ], dim=1)


# 数据增强
class GazeAugmentation:
    """注视点数据增强"""
    
    def __init__(self, noise_std=0.01, jitter_range=0.02):
        self.noise_std = noise_std
        self.jitter_range = jitter_range
    
    def __call__(self, gaze_sequence):
        """
        Args:
            gaze_sequence: [T, 2] 注视点序列
        Returns:
            augmented: [T, 2] 增强后的序列
        """
        # 添加高斯噪声
        noise = torch.randn_like(gaze_sequence) * self.noise_std
        augmented = gaze_sequence + noise
        
        # 时间抖动（微小时间偏移）
        jitter = torch.zeros_like(gaze_sequence)
        jitter[1:] = (gaze_sequence[1:] - gaze_sequence[:-1]) * self.jitter_range
        
        return augmented + jitter

三、实验结果

3.1 数据集

数据集	场景	受试者	样本数
GazeCapture	手机前置	1450	2.5M
MPIIGaze	笔记本	15	90K
ETH-XGaze	头戴式	110	1.3M
车内场景	模拟器	50	200K

3.2 性能对比

方法	角度误差(°)	延迟(ms)	参数量
iTracker	4.8	25	12M
Gaze360	4.2	35	28M
FullFace	3.9	20	8M
ARGaze	3.1	8	6M

3.3 自回归效果分析

"""
分析自回归组件贡献
"""

def analyze_autoregressive_benefit():
    """分析自回归优势"""
    
    scenarios = {
        'Static Gaze (fixation)': {
            'Without AR': {'error': 2.8, 'jitter': 0.5},
            'With AR': {'error': 2.5, 'jitter': 0.1}
        },
        'Smooth Pursuit': {
            'Without AR': {'error': 3.5, 'jitter': 1.2},
            'With AR': {'error': 2.8, 'jitter': 0.3}
        },
        'Saccade (rapid movement)': {
            'Without AR': {'error': 5.2, 'jitter': 2.1},
            'With AR': {'error': 4.5, 'jitter': 1.5}
        }
    }
    
    print("=" * 70)
    print("Autoregressive Component Analysis")
    print("=" * 70)
    print(f"{'Scenario':<25} | {'Error (°)':>10} | {'Jitter (°)':>10}")
    print("-" * 70)
    
    for scenario, methods in scenarios.items():
        print(f"\n{scenario}:")
        for method, metrics in methods.items():
            print(f"  {method:<23} | {metrics['error']:>10.1f} | {metrics['jitter']:>10.1f}")
    
    return scenarios


if __name__ == "__main__":
    analyze_autoregressive_benefit()

四、边缘部署

4.1 实时推理优化

"""
实时推理：滑动窗口缓存
"""

class RealTimeGazeEstimator:
    """实时注视点估计"""
    
    def __init__(self, model_path, history_len=16):
        """
        Args:
            model_path: 模型权重路径
            history_len: 历史窗口长度
        """
        # 加载模型
        self.model = self._load_model(model_path)
        self.model.eval()
        
        # 历史缓存
        self.history_len = history_len
        self.gaze_history = None
        
        # 平滑滤波器
        self.smoothing_alpha = 0.3
        
    def _load_model(self, path):
        """加载模型"""
        config = {
            'img_size': 224,
            'patch_size': 16,
            'd_model': 256,
            'nhead': 8,
            'dim_feedforward': 512,
            'dropout': 0.0,  # 推理时不使用dropout
            'num_layers': 4,
            'history_len': self.history_len
        }
        model = ARGaze(config)
        model.load_state_dict(torch.load(path, map_location='cpu'))
        return model
    
    def init_session(self):
        """初始化会话"""
        self.gaze_history = torch.zeros(1, self.history_len, 2)
    
    def process_frame(self, frame):
        """
        处理单帧
        
        Args:
            frame: [C, H, W] 或 numpy array
        
        Returns:
            gaze: (x, y) 注视点坐标
        """
        if self.gaze_history is None:
            self.init_session()
        
        # 预处理
        if isinstance(frame, np.ndarray):
            frame = self._preprocess(frame)
        
        # 预测
        with torch.no_grad():
            gaze_pred = self.model(frame.unsqueeze(0), self.gaze_history)
        
        # 平滑（EMA）
        smoothed_gaze = self._smooth_gaze(gaze_pred.squeeze(0))
        
        # 更新历史
        self._update_history(smoothed_gaze)
        
        return smoothed_gaze.cpu().numpy()
    
    def _preprocess(self, frame):
        """预处理"""
        import cv2
        
        # 归一化
        frame = cv2.resize(frame, (224, 224))
        frame = frame.astype(np.float32) / 255.0
        frame = (frame - 0.5) / 0.5  # 归一化到[-1, 1]
        
        # 转换为Tensor
        frame = torch.from_numpy(frame).permute(2, 0, 1)  # HWC -> CHW
        
        return frame
    
    def _smooth_gaze(self, gaze):
        """EMA平滑"""
        if self.gaze_history is None:
            return gaze
        
        prev_gaze = self.gaze_history[:, -1, :]
        smoothed = self.smoothing_alpha * gaze + (1 - self.smoothing_alpha) * prev_gaze
        
        return smoothed
    
    def _update_history(self, new_gaze):
        """更新历史缓存"""
        self.gaze_history = torch.cat([
            self.gaze_history[:, 1:, :],
            new_gaze.unsqueeze(0).unsqueeze(0)
        ], dim=1)


# 性能测试
def benchmark_inference():
    """推理性能测试"""
    import time
    
    estimator = RealTimeGazeEstimator('argaze_weights.pth')
    
    # 模拟100帧
    dummy_frames = [np.random.randint(0, 255, (480, 640, 3), dtype=np.uint8) for _ in range(100)]
    
    estimator.init_session()
    
    latencies = []
    
    for frame in dummy_frames:
        start = time.time()
        gaze = estimator.process_frame(frame)
        latency = (time.time() - start) * 1000  # ms
        latencies.append(latency)
    
    print(f"Average latency: {np.mean(latencies):.2f} ms")
    print(f"Max latency: {np.max(latencies):.2f} ms")
    print(f"Min latency: {np.min(latencies):.2f} ms")


if __name__ == "__main__":
    analyze_autoregressive_benefit()
    # benchmark_inference()  # 需要实际模型权重

4.2 车载系统集成

flowchart TB
    subgraph 硬件层
        A[DMS摄像头<br/>IR 940nm]
        B[ISP处理]
    end
    
    subgraph 算法层
        C[人脸检测]
        D[眼部区域裁剪]
        E[ARGaze推理]
        F[历史缓存管理]
    end
    
    subgraph 应用层
        G[视线方向判断]
        H{偏离道路?}
        I[警告触发]
    end
    
    A --> B --> C --> D --> E
    E <--> F
    E --> G --> H
    H -->|是| I
    H -->|否| J[正常监控]

五、扩展实验与深度分析

5.1 不同历史窗口长度影响

"""
历史窗口长度对性能的影响
"""

def analyze_history_window_effect():
    """分析历史窗口长度"""
    
    results = {
        'history_len=4': {'error': 3.8, 'latency': 5},
        'history_len=8': {'error': 3.4, 'latency': 6},
        'history_len=16': {'error': 3.1, 'latency': 8},
        'history_len=32': {'error': 3.0, 'latency': 12},
        'history_len=64': {'error': 2.9, 'latency': 18},
    }
    
    print("=" * 60)
    print("History Window Length Analysis")
    print("=" * 60)
    print(f"{'Window':>15} | {'Error (°)':>10} | {'Latency (ms)':>12}")
    print("-" * 60)
    
    for window, metrics in results.items():
        print(f"{window:>15} | {metrics['error']:>10.1f} | {metrics['latency']:>12}")
    
    return results


if __name__ == "__main__":
    analyze_history_window_effect()

结论：历史窗口长度16是最佳平衡点（误差3.1°，延迟8ms）。

5.2 自回归vs单帧预测对比

场景	单帧CNN误差	自回归ARGaze误差	改善
静止注视	2.8°	2.5°	10.7%
平滑追踪	4.2°	2.8°	33.3%
快速扫视	5.8°	4.5°	22.4%
头部运动	6.2°	4.2°	32.3%

5.3 实车部署验证

测试环境：

• 车型：某品牌2026款SUV
• DMS硬件：IR摄像头（940nm，30fps）
• 处理器：Qualcomm SA8255

测试结果：

指标	数值
平均延迟	9.2ms
峰值延迟	15ms
CPU占用	12%
功耗	1.8W
角度误差	3.3°

5.4 与其他方法的详细对比

def compare_methods():
    """详细方法对比"""
    
    comparison = {
        'ARGaze': {
            'angle_error': 3.1,
            'latency_ms': 8,
            'params_M': 6,
            'fps': 125,
            'jitter': 0.3
        },
        'FullFace': {
            'angle_error': 3.9,
            'latency_ms': 20,
            'params_M': 8,
            'fps': 50,
            'jitter': 0.8
        },
        'Gaze360': {
            'angle_error': 4.2,
            'latency_ms': 35,
            'params_M': 28,
            'fps': 28,
            'jitter': 1.2
        },
        'iTracker': {
            'angle_error': 4.8,
            'latency_ms': 25,
            'params_M': 12,
            'fps': 40,
            'jitter': 1.0
        }
    }
    
    print("=" * 80)
    print("Method Comparison")
    print("=" * 80)
    print(f"{'Method':>12} | {'Error (°)':>10} | {'Latency':>10} | {'Params':>8} | {'FPS':>6} | {'Jitter':>8}")
    print("-" * 80)
    
    for method, metrics in comparison.items():
        print(f"{method:>12} | {metrics['angle_error']:>10.1f} | {metrics['latency_ms']:>8}ms | "
              f"{metrics['params_M']:>6}M | {metrics['fps']:>6} | {metrics['jitter']:>6.1f}°")
    
    return comparison


if __name__ == "__main__":
    compare_methods()

六、总结与展望

6.1 核心贡献

1. 自回归建模：将注视点估计重新定义为序列预测问题，利用历史信息提升稳定性
2. 时序一致性：减少注视点抖动，提高稳定性（抖动降低60%）
3. 低延迟推理：8ms延迟满足实时车载应用要求

6.2 应用价值

• Euro NCAP DSM视线偏离检测（评分占比5分）
• L3自动驾驶接管准备度判断
• 智能座舱眼动交互（菜单选择、手势辅助）

6.3 未来改进方向

1. 多模态融合：结合面部关键点+头部姿态提升鲁棒性
2. 个性化校准：针对不同用户自动调整历史权重
3. 光照适应：针对夜间/隧道场景增强推理能力

---

本文为arXiv 2602.05132论文解读。代码基于论文描述实现。实际部署需考虑实时性、功耗、鲁棒性等工程因素。