GazeSymCAT跨注意力Transformer：极端头部姿态下的鲁棒视线估计

来源：Oxford Academic (March 2025)
链接：https://academic.oup.com/jcde/article/12/3/115/8003774

核心创新

对称跨注意力架构解决极端头部姿态问题：

• ETH-XGaze准确率：SOTA
• 极端姿态角度：支持±60°偏转
• 跨数据集泛化：无需微调

技术背景

极端头部姿态挑战

挑战	传统方法局限	GazeSymCAT解决
大角度偏转	特征丢失	对称注意力
自遮挡	单眼不可见	跨眼注意力
数据不均衡	极端样本少	数据增强

方法架构

import torch
import torch.nn as nn
import torch.nn.functional as F

class GazeSymCAT(nn.Module):
    """
    GazeSymCAT: 对称跨注意力视线估计
    
    核心创新：
    1. 双眼对称注意力
    2. 跨眼特征交互
    3. 极端姿态鲁棒
    """
    
    def __init__(self, embed_dim=256, num_heads=8):
        super().__init__()
        
        # 左右眼特征提取器（共享权重）
        self.eye_encoder = EyeEncoder()
        
        # 跨注意力模块
        self.cross_attention = CrossAttention(embed_dim, num_heads)
        
        # 对称注意力（左右眼交换）
        self.symmetric_attention = SymmetricAttention(embed_dim, num_heads)
        
        # 头部姿态编码
        self.head_pose_encoder = HeadPoseEncoder()
        
        # 视线预测头
        self.gaze_head = nn.Linear(embed_dim * 2 + 128, 3)
        
    def forward(self, left_eye, right_eye, head_pose):
        """
        Args:
            left_eye: (B, 3, 64, 32) 左眼图像
            right_eye: (B, 3, 64, 32) 右眼图像
            head_pose: (B, 3) 头部姿态（pitch, yaw, roll）
        
        Returns:
            gaze_vector: (B, 3) 视线方向
        """
        # 提取双眼特征
        left_feat = self.eye_encoder(left_eye)   # (B, 256)
        right_feat = self.eye_encoder(right_eye) # (B, 256)
        
        # 跨注意力（左眼关注右眼）
        left_cross = self.cross_attention(left_feat, right_feat)
        
        # 跨注意力（右眼关注左眼）
        right_cross = self.cross_attention(right_feat, left_feat)
        
        # 对称注意力（融合）
        fused = self.symmetric_attention(left_cross, right_cross)
        
        # 头部姿态编码
        pose_feat = self.head_pose_encoder(head_pose)  # (B, 128)
        
        # 拼接并预测
        combined = torch.cat([fused, pose_feat], dim=-1)
        gaze = self.gaze_head(combined)
        
        return F.normalize(gaze, dim=-1)


class CrossAttention(nn.Module):
    """
    跨眼注意力模块
    
    一只眼的特征关注另一只眼
    """
    
    def __init__(self, embed_dim, num_heads):
        super().__init__()
        self.attention = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
        
    def forward(self, query, reference):
        """
        Args:
            query: (B, D) 查询眼特征
            reference: (B, D) 参考眼特征
        
        Returns:
            attended: (B, D) 注意力输出
        """
        # 扩展维度
        query = query.unsqueeze(1)     # (B, 1, D)
        reference = reference.unsqueeze(1)  # (B, 1, D)
        
        # 跨注意力
        attended, _ = self.attention(query, reference, reference)
        
        return attended.squeeze(1)  # (B, D)


class SymmetricAttention(nn.Module):
    """
    对称注意力模块
    
    左右眼特征对称融合
    """
    
    def __init__(self, embed_dim, num_heads):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
        self.norm = nn.LayerNorm(embed_dim)
        
    def forward(self, left_feat, right_feat):
        """
        Args:
            left_feat: (B, D)
            right_feat: (B, D)
        
        Returns:
            fused: (B, D)
        """
        # 拼接为序列
        sequence = torch.stack([left_feat, right_feat], dim=1)  # (B, 2, D)
        
        # 自注意力
        attended, _ = self.self_attn(sequence, sequence, sequence)
        
        # 残差连接
        sequence = self.norm(sequence + attended)
        
        # 池化
        fused = sequence.mean(dim=1)  # (B, D)
        
        return fused

实验结果

ETH-XGaze性能

方法	正常姿态误差	极端姿态误差	平均误差
Full Face CNN	4.2°	8.5°	5.8°
Appearance Gaze	3.8°	7.2°	5.1°
GazeOnce	3.5°	6.1°	4.5°
GazeSymCAT	3.2°	4.8°	3.8°

极端姿态定义

姿态类型	Yaw范围	Pitch范围	传统方法性能
正常	±15°	±10°	正常
中等	±30°	±20°	下降10-15%
极端	±60°	±40°	下降30-50%

IMS应用启示

车辆座舱特殊场景

class InVehicleGazeEstimator:
    """
    车内视线估计器
    
    特殊挑战：
    1. 驾驶员频繁转头（检查后视镜）
    2. 遮挡（墨镜/口罩）
    3. 光照变化（进隧道）
    """
    
    def __init__(self):
        self.model = GazeSymCAT()
        
    def estimate_with_occlusion_handling(self, left_eye, right_eye, head_pose):
        """
        处理遮挡场景
        
        策略：
        - 单眼可用时，依赖跨注意力从另一眼推断
        """
        # 检测眼部遮挡
        left_visible = self._check_eye_visibility(left_eye)
        right_visible = self._check_eye_visibility(right_eye)
        
        if left_visible and right_visible:
            # 双眼可用
            gaze = self.model(left_eye, right_eye, head_pose)
        elif left_visible:
            # 仅左眼可用，复制到右眼
            gaze = self.model(left_eye, left_eye, head_pose)
        elif right_visible:
            # 仅右眼可用
            gaze = self.model(right_eye, right_eye, head_pose)
        else:
            # 双眼不可见，依赖头部姿态推断
            gaze = self._estimate_from_head_pose(head_pose)
        
        return gaze
    
    def _check_eye_visibility(self, eye_image):
        """检查眼部是否可见"""
        # 简化：基于亮度方差
        brightness_var = eye_image.var()
        return brightness_var > 0.01

Euro NCAP合规

要求	GazeSymCAT	状态
角度误差<5°	3.8°	✅
极端姿态支持	±60°	✅
实时性	需优化	⚠️

参考文献

1. GazeSymCAT: A symmetric cross-attention transformer for robust gaze estimation. Oxford Academic (2025).
2. ETH-XGaze Dataset.

---

总结： GazeSymCAT通过跨注意力机制解决了极端头部姿态下的视线估计问题，适合车内驾驶员频繁转头的场景。