GazeTR Transformer架构详解：Pure与Hybrid的较量

引言：Transformer进军视线估计

2021年，Vision Transformer（ViT）在ImageNet上超越ResNet，开启了Transformer在计算机视觉的新时代。但Transformer在视线估计领域的应用仍是空白。

GazeTR（ICPR 2022）是首个系统探索Transformer在视线估计中应用的工作，提出了两种架构：

• GazeTR-Pure：纯Transformer架构
• GazeTR-Hybrid：CNN + Transformer混合架构

实验结果颠覆直觉：Hybrid以更少参数实现更高精度。

---

一、GazeTR-Pure：纯Transformer架构

1.1 架构设计

GazeTR-Pure完全遵循ViT设计：

输入图像 (224×224)
    ↓
┌─────────────────────────────────┐
│ Patch Embedding                 │
│ - 16×16 patches                 │
│ - 196 tokens + 1 CLS token      │
│ - Positional Encoding           │
└─────────────────────────────────┘
    ↓
┌─────────────────────────────────┐
│ Transformer Encoder × L层       │
│ - Multi-Head Self-Attention     │
│ - Layer Norm + MLP              │
│ - Residual Connection           │
└─────────────────────────────────┘
    ↓
┌─────────────────────────────────┐
│ CLS Token → Gaze Head           │
│ - Linear → [pitch, yaw]         │
└─────────────────────────────────┘

1.2 Patch Embedding

import torch
import torch.nn as nn

class PatchEmbedding(nn.Module):
    def __init__(self, img_size=224, patch_size=16, in_channels=3, embed_dim=768):
        super().__init__()
        self.num_patches = (img_size // patch_size) ** 2  # 196
        self.proj = nn.Conv2d(in_channels, embed_dim, 
                              kernel_size=patch_size, stride=patch_size)
        
    def forward(self, x):
        # x: [B, 3, 224, 224]
        x = self.proj(x)  # [B, 768, 14, 14]
        x = x.flatten(2).transpose(1, 2)  # [B, 196, 768]
        return x

class GazeTRPure(nn.Module):
    def __init__(self, img_size=224, patch_size=16, embed_dim=768, 
                 depth=12, num_heads=12):
        super().__init__()
        
        # Patch Embedding
        self.patch_embed = PatchEmbedding(img_size, patch_size, 3, embed_dim)
        
        # CLS Token + Positional Encoding
        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
        self.pos_embed = nn.Parameter(torch.zeros(1, 197, embed_dim))
        
        # Transformer Encoder
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embed_dim, 
            nhead=num_heads,
            dim_feedforward=embed_dim * 4,
            dropout=0.1
        )
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=depth)
        
        # Gaze Regression Head
        self.gaze_head = nn.Linear(embed_dim, 2)  # [pitch, yaw]
        
    def forward(self, x):
        B = x.size(0)
        
        # Patch Embedding
        x = self.patch_embed(x)  # [B, 196, 768]
        
        # Add CLS token
        cls_tokens = self.cls_token.expand(B, -1, -1)
        x = torch.cat([cls_tokens, x], dim=1)  # [B, 197, 768]
        
        # Add Positional Encoding
        x = x + self.pos_embed
        
        # Transformer Encoder
        x = self.transformer(x)
        
        # Gaze Regression
        gaze = self.gaze_head(x[:, 0])  # Use CLS token
        
        return gaze

1.3 参数量分析

组件	参数量
Patch Embedding	0.6M
Positional Encoding	0.15M
Transformer (12层)	85.8M
Gaze Head	0.002M
总计	86.6M

---

二、GazeTR-Hybrid：混合架构

2.1 设计动机

问题：纯Transformer缺乏CNN的局部特征提取能力

解决方案：保留CNN作为特征提取器，Transformer增强全局建模

2.2 架构设计

输入图像 (224×224)
    ↓
┌─────────────────────────────────┐
│ ResNet-18 Backbone              │
│ - Conv1 → 64通道                │
│ - Stage1-4 → [64, 128, 256, 512]│
│ - 输出: [B, 512, 7, 7]          │
└─────────────────────────────────┘
    ↓
┌─────────────────────────────────┐
│ Feature Flattening              │
│ - 49 tokens (7×7)               │
│ - Positional Encoding           │
└─────────────────────────────────┘
    ↓
┌─────────────────────────────────┐
│ Transformer Encoder × 2层       │
│ - Lightweight Self-Attention    │
│ - Global Context Modeling       │
└─────────────────────────────────┘
    ↓
┌─────────────────────────────────┐
│ Gaze Regression Head            │
│ - Global Average Pooling        │
│ - FC → [pitch, yaw]             │
└─────────────────────────────────┘

2.3 代码实现

import torchvision.models as models

class GazeTRHybrid(nn.Module):
    def __init__(self, embed_dim=512, depth=2, num_heads=8):
        super().__init__()
        
        # ResNet-18 Backbone
        resnet = models.resnet18(pretrained=True)
        self.backbone = nn.Sequential(*list(resnet.children())[:-2])
        # 输出: [B, 512, 7, 7]
        
        # Positional Encoding
        self.pos_embed = nn.Parameter(torch.zeros(1, 49, embed_dim))
        
        # Transformer Encoder (轻量级)
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embed_dim,
            nhead=num_heads,
            dim_feedforward=embed_dim * 2,
            dropout=0.1
        )
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=depth)
        
        # Gaze Regression Head
        self.gaze_head = nn.Sequential(
            nn.Linear(embed_dim, 256),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(256, 2)  # [pitch, yaw]
        )
        
    def forward(self, x):
        B = x.size(0)
        
        # CNN Feature Extraction
        features = self.backbone(x)  # [B, 512, 7, 7]
        
        # Flatten to tokens
        tokens = features.flatten(2).transpose(1, 2)  # [B, 49, 512]
        
        # Add Positional Encoding
        tokens = tokens + self.pos_embed
        
        # Transformer Encoder
        tokens = self.transformer(tokens)
        
        # Global Average Pooling
        x = tokens.mean(dim=1)  # [B, 512]
        
        # Gaze Regression
        gaze = self.gaze_head(x)
        
        return gaze

2.4 参数量分析

组件	参数量
ResNet-18 Backbone	11.2M
Positional Encoding	0.025M
Transformer (2层)	2.1M
Gaze Head	0.13M
总计	13.5M

对比：GazeTR-Hybrid参数量仅为GazeTR-Pure的15.6%。

---

三、实验对比

3.1 数据集性能

ETH-XGaze数据集：

方法	MAE（角误差）	参数量
FullFace	6.53°	196.6M
RT-GENE	6.02°	45.2M
GazeTR-Pure	5.89°	86.6M
GazeTR-Hybrid	5.33°	13.5M

MPIIFaceGaze数据集：

方法	MAE
FullFace	4.95°
Dilated-Net	4.78°
GazeTR-Pure	4.52°
GazeTR-Hybrid	4.06°

Gaze360数据集：

方法	MAE
L2CS-Net	9.46°
GazeCapsNet	5.10°
GazeTR-Hybrid	4.50°

3.2 跨数据集泛化

训练→测试：

训练数据	测试数据	GazeTR-Pure	GazeTR-Hybrid
ETH-XGaze	MPIIFaceGaze	6.23°	5.94°
Gaze360	ETH-XGaze	6.82°	5.87°
ETH-XGaze	Gaze360	7.45°	6.82°

结论：GazeTR-Hybrid泛化能力更强。

---

四、消融实验

4.1 Transformer层数影响

层数	ETH-XGaze MAE	参数量
1	5.67°	12.4M
2	5.33°	13.5M
4	5.41°	15.7M
6	5.58°	17.9M

结论：2层Transformer最佳，更多层反而过拟合。

4.2 Self-Attention可视化

import matplotlib.pyplot as plt
import seaborn as sns

def visualize_attention(model, image, layer_idx=0):
    """
    可视化Self-Attention权重
    """
    model.eval()
    with torch.no_grad():
        # 前向传播，获取attention weights
        features = model.backbone(image)
        tokens = features.flatten(2).transpose(1, 2)
        
        # 获取第layer_idx层的attention
        attn_weights = model.transformer.layers[layer_idx]
                                         .self_attn
                                         .get_attention_weights(tokens)
    
    # 可视化
    plt.figure(figsize=(10, 8))
    sns.heatmap(attn_weights[0].mean(dim=0).cpu().numpy(),
                cmap='viridis', 
                xticklabels=5, yticklabels=5)
    plt.xlabel('Token Index')
    plt.ylabel('Token Index')
    plt.title(f'Self-Attention Weights (Layer {layer_idx})')
    plt.savefig(f'attention_layer_{layer_idx}.png', dpi=150)
    plt.show()

# 使用示例
image = load_image('driver.jpg')
visualize_attention(model, image, layer_idx=0)

4.3 CNN骨干网络对比

Backbone	ETH-XGaze MAE	参数量	推理时间
ResNet-18	5.33°	13.5M	25ms
ResNet-50	5.21°	23.8M	35ms
MobileNet v2	5.68°	9.2M	18ms
EfficientNet-B0	5.45°	5.3M	15ms

结论：ResNet-18在精度与速度间取得最佳平衡。

---

五、与GazeCapsNet对比

维度	GazeCapsNet	GazeTR-Hybrid
核心机制	Capsule Network	Transformer
参数量	11.7M	13.5M
ETH-XGaze MAE	5.10°	5.33°
MPIIFaceGaze MAE	4.06°	4.06°
推理时间	20ms	25ms
优势	空间关系建模	全局上下文建模
劣势	训练复杂	需要大数据量预训练

选型建议：

• 车载DMS：GazeCapsNet（更快、更轻量）
• 科研实验：GazeTR-Hybrid（更强全局建模）

---

六、嵌入式部署

6.1 模型量化

import torch.quantization as quant

# 动态量化
model_quantized = quant.quantize_dynamic(
    model,
    {nn.Linear, nn.TransformerEncoderLayer},
    dtype=torch.qint8
)

# 性能对比
# FP32: 25ms, 13.5M params
# INT8: 15ms, 3.4M params

6.2 TensorRT加速

import torch.onnx
import tensorrt as trt

# 导出ONNX
torch.onnx.export(
    model,
    torch.randn(1, 3, 224, 224),
    'gazetr_hybrid.onnx',
    opset_version=11
)

# TensorRT优化
# FP32: 25ms
# FP16: 15ms
# INT8: 10ms (需校准)

---

七、总结

7.1 核心结论

结论	说明
Hybrid > Pure	CNN + Transformer混合架构优于纯Transformer
少即是多	2层Transformer足够，更多层反而过拟合
参数效率	13.5M参数实现SOTA精度
泛化性强	跨数据集性能优于Pure

7.2 GazeTR vs GazeCapsNet

场景	推荐
量产车载DMS	GazeCapsNet
科研实验	GazeTR-Hybrid
边缘设备	GazeCapsNet量化版
高精度需求	GazeTR-Hybrid + ResNet-50

---

参考文献

1. Cheng, Y., & Lu, F. “Gaze Estimation using Transformer.” ICPR, 2022.
2. Dosovitskiy, A., et al. “An Image is Worth 16x16 Words.” ICLR, 2021.
3. Zhang, X., et al. “ETH-XGaze: A Large Scale Dataset.” ECCV, 2020.

---

本文是IMS视线估计算法系列文章之一，上一篇：GazeCapsNet详解