GazeGene：大规模合成视线数据集与3D眼球标注（CVPR 2025）

GazeGene：大规模合成视线数据集与3D眼球标注（CVPR 2025）论文解读与代码复现

核心贡献：首个提供完整3D眼球结构标注的合成视线数据集，突破现有数据集标注错误和结构缺失的瓶颈。

一、论文信息

项目	内容
标题	GazeGene: Large-scale Synthetic Gaze Dataset with 3D Eyeball Annotations
作者	Yiwei Bao, Zhiming Wang, Feng Lu (Beihang University)
会议	IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025)
链接	https://openaccess.thecvf.com/content/CVPR2025/papers/Bao_GazeGene_Large-scale_Synthetic_Gaze_Dataset_with_3D_Eyeball_Annotations_CVPR_2025_paper.pdf
数据集	https://huggingface.co/datasets/vigil1917/GazeGene

---

二、研究背景与动机

2.1 现有视线估计数据集的三大问题

问题	具体表现	影响
标注错误	MPIIGaze、Ethereal 等数据集存在 5-15% 标注偏差	模型泛化能力受限
结构缺失	仅提供视线向量，缺少眼球、瞳孔、虹膜等3D结构	无法支撑精细任务
分布单一	头部姿态、光照、表情多样性不足	跨域性能下降

2.2 合成数据的优势

真实数据                    合成数据
├── 标注成本高              ├── 标注零成本（自动生成）
├── 隐私合规风险            ├── 无隐私问题（虚拟人物）
├── 场景覆盖受限            ├── 无限场景扩展
└── 标注一致性差            └── 完美标注一致性

---

三、数据集核心特性

3.1 规模与配置

指标	数值
总样本数	1,008,000 张图像
受试者	56 人（不同种族/性别）
相机数量	9 个（多视角覆盖）
每人帧数	2000 帧
分辨率	高分辨率面部裁剪

3.2 标注内容（首次提供）

传统数据集 vs GazeGene

标注类型	MPIIGaze	ETH-XGaze	GazeGene
视线向量	✅	✅	✅
头部姿态	✅	✅	✅
眼球中心 3D	❌	❌	✅ 首次
瞳孔中心 2D/3D	❌	❌	✅ 首次
虹膜轮廓 100 点	❌	❌	✅ 首次
视轴/光轴	❌	❌	✅ 首次
角膜参数	❌	❌	✅ 首次

3.3 数据分布多样性

# 多样性维度
diversity_dimensions = {
    "表情": ["中性", "微笑", "皱眉", "惊讶", "愤怒"],
    "眨眼": ["睁眼", "半闭", "闭眼"],
    "瞳孔大小": ["缩小", "正常", "放大"],
    "光照温度": ["暖光(3000K)", "中性(5000K)", "冷光(7000K)"],
    "光照强度": ["暗(10 lux)", "正常(100 lux)", "亮(500 lux)"],
    "光照方向": ["正面", "左侧", "右侧", "上方", "下方"],
    "头部姿态": ["俯仰", "偏航", "滚转"] * 平滑分布,
    "视线分布": ["中心", "边缘"] * 大范围覆盖
}

---

四、技术方法：基于 Metahuman 的合成管道

4.1 整体流程

graph LR
    A[Metahuman<br/>角色创建] --> B[动画驱动<br/>视线控制]
    B --> C[多相机渲染<br/>9视角]
    C --> D[标注生成<br/>3D结构提取]
    D --> E[归一化处理<br/>统一坐标系]
    E --> F[数据集输出]

4.2 Metahuman 视线控制

# Unreal Engine + Metahuman 视线控制
import unreal

class GazeController:
    """Metahuman 视线控制器"""
    
    def __init__(self, metahuman_actor):
        self.metahuman = metahuman_actor
        self.left_eye = metahuman_actor.get_component_by_name("LeftEye")
        self.right_eye = metahuman_actor.get_component_by_name("RightEye")
    
    def set_gaze_target(self, target_location):
        """
        设置视线目标
        
        Args:
            target_location: 3D目标点 (x, y, z)
        """
        # 计算视线方向
        head_location = self.metahuman.get_actor_location()
        gaze_direction = (target_location - head_location).normalized()
        
        # 驱动眼球旋转
        self.left_eye.set_world_rotation(
            self._calculate_eye_rotation(gaze_direction, is_left=True)
        )
        self.right_eye.set_world_rotation(
            self._calculate_eye_rotation(gaze_direction, is_left=False)
        )
    
    def _calculate_eye_rotation(self, gaze_direction, is_left):
        """计算眼球旋转角度"""
        # 考虑 kappa 角（光轴与视轴夹角）
        kappa_angle = self.metahuman.get_kappa_angle(is_left)
        
        # 修正视线方向
        visual_axis = gaze_direction.rotate_around_axis(
            axis=(0, 1, 0),
            angle=kappa_angle
        )
        
        return visual_axis.to_rotator()

4.3 标注生成算法

import numpy as np

class GazeAnnotator:
    """视线标注生成器"""
    
    def __init__(self, camera_params, subject_params):
        self.cameras = camera_params
        self.subject = subject_params
    
    def generate_3d_annotations(self, frame_id):
        """
        生成3D眼球标注
        
        Returns:
            dict: 包含眼球中心、瞳孔中心、虹膜轮廓等
        """
        # 1. 眼球中心（基于解剖学模型）
        eyeball_center_L = self.subject['eyecenter_L']  # 左眼球中心
        eyeball_center_R = self.subject['eyecenter_R']  # 右眼球中心
        eyeball_radius = self.subject['eyeball_radius']
        
        # 2. 瞳孔中心（考虑瞳孔大小变化）
        pupil_center_L = self._calculate_pupil_center(
            eyeball_center_L, 
            self.subject['UVRadius'],  # 瞳孔大小
            self.current_gaze_L
        )
        
        # 3. 虹膜轮廓（100点）
        iris_contour_L = self._generate_iris_contour(
            pupil_center_L,
            self.subject['iris_radius'],
            num_points=100
        )
        
        # 4. 视轴与光轴
        visual_axis_L = self._calculate_visual_axis(pupil_center_L)
        optic_axis_L = self._calculate_optic_axis(eyeball_center_L)
        
        return {
            'eyeball_center_3D': np.array([eyeball_center_L, eyeball_center_R]),
            'pupil_center_3D': np.array([pupil_center_L, pupil_center_R]),
            'iris_mesh_3D': np.array([iris_contour_L, iris_contour_R]),
            'visual_axis': np.array([visual_axis_L, visual_axis_R]),
            'optic_axis': np.array([optic_axis_L, optic_axis_R])
        }
    
    def _generate_iris_contour(self, center, radius, num_points=100):
        """生成虹膜轮廓3D点"""
        angles = np.linspace(0, 2*np.pi, num_points)
        
        # 虹膜在眼球表面的投影
        contour = np.zeros((num_points, 3))
        for i, angle in enumerate(angles):
            # 球面坐标转换
            theta = angle  # 方位角
            phi = np.arcsin(radius / self.subject['eyeball_radius'])  # 极角
            
            contour[i] = [
                center[0] + radius * np.sin(phi) * np.cos(theta),
                center[1] + radius * np.sin(phi) * np.sin(theta),
                center[2] + radius * np.cos(phi)
            ]
        
        return contour

---

五、代码复现：数据加载与使用

5.1 数据集下载

# Hugging Face 下载
huggingface-cli download vigil1917/GazeGene \
  --repo-type dataset \
  --local-dir ./GazeGene \
  --local-dir-use-symlinks False

# 目录结构
GazeGene_normalized/
├── Example.py
├── subject1/
│   ├── labels_normalized/
│   │   ├── gaze_label_camera0_normalized.pkl
│   │   └── complex_camera0_normalized.pkl
│   ├── imgs_normalized/
│   │   └── subject1_0000.jpg
│   ├── camera_info.pkl
│   └── subject_label.pkl
└── subject56/

5.2 Python 数据加载器

import pickle
import numpy as np
from pathlib import Path
import cv2

class GazeGeneDataset:
    """GazeGene 数据集加载器"""
    
    def __init__(self, data_root):
        self.data_root = Path(data_root)
        self.subjects = sorted([d.name for d in self.data_root.iterdir() 
                                if d.is_dir() and d.name.startswith('subject')])
    
    def load_sample(self, subject_id, camera_id, frame_id):
        """
        加载单个样本
        
        Args:
            subject_id: 受试者ID (1-56)
            camera_id: 相机ID (0-8)
            frame_id: 帧ID (0-1999)
        
        Returns:
            dict: 包含图像和所有标注
        """
        subject_dir = self.data_root / f"subject{subject_id}"
        
        # 1. 加载图像
        img_path = subject_dir / "imgs_normalized" / f"subject{subject_id}_{frame_id:04d}.jpg"
        image = cv2.imread(str(img_path))
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        
        # 2. 加载视线标注
        gaze_file = subject_dir / "labels_normalized" / f"gaze_label_camera{camera_id}_normalized.pkl"
        with open(gaze_file, 'rb') as f:
            gaze_data = pickle.load(f)
        
        # 3. 加载3D眼球标注
        complex_file = subject_dir / "labels_normalized" / f"complex_camera{camera_id}_normalized.pkl"
        with open(complex_file, 'rb') as f:
            complex_data = pickle.load(f)
        
        # 4. 提取当前帧标注
        sample = {
            'image': image,
            'gaze_vector': gaze_data['gaze_C'][frame_id],
            'visual_axis_left': gaze_data['visual_axis_L'][frame_id],
            'visual_axis_right': gaze_data['visual_axis_R'][frame_id],
            'head_pose': gaze_data['head_R_mat'][frame_id],
            'eyeball_center_2D': complex_data['eyeball_center_2D'][frame_id],
            'eyeball_center_3D': complex_data['eyeball_center_3D'][frame_id],
            'pupil_center_2D': complex_data['pupil_center_2D'][frame_id],
            'pupil_center_3D': complex_data['pupil_center_3D'][frame_id],
            'iris_contour_2D': complex_data['iris_mesh_2D'][frame_id],
            'iris_contour_3D': complex_data['iris_mesh_3D'][frame_id]
        }
        
        return sample
    
    def __len__(self):
        return len(self.subjects) * 9 * 2000  # 56 * 9 * 2000 = 1,008,000
    
    def __getitem__(self, idx):
        subject_id = (idx // (9 * 2000)) + 1
        remaining = idx % (9 * 2000)
        camera_id = remaining // 2000
        frame_id = remaining % 2000
        
        return self.load_sample(subject_id, camera_id, frame_id)


# 使用示例
if __name__ == "__main__":
    dataset = GazeGeneDataset("./GazeGene_normalized")
    
    # 加载第一个样本
    sample = dataset.load_sample(subject_id=44, camera_id=0, frame_id=0)
    
    print(f"图像尺寸: {sample['image'].shape}")
    print(f"视线向量: {sample['gaze_vector']}")
    print(f"左眼球中心2D: {sample['eyeball_center_2D'][0]}")
    print(f"虹膜轮廓点数: {sample['iris_contour_2D'][0].shape[0]}")

5.3 可视化工具

import matplotlib.pyplot as plt
from matplotlib.patches import Ellipse, Circle

def visualize_gaze_sample(sample):
    """可视化视线标注"""
    fig, ax = plt.subplots(1, 1, figsize=(10, 8))
    
    # 显示图像
    ax.imshow(sample['image'])
    
    # 绘制眼球中心
    for i, (eye_center, color) in enumerate([
        (sample['eyeball_center_2D'][0], 'red'),  # 左眼
        (sample['eyeball_center_2D'][1], 'blue')   # 右眼
    ]):
        ax.plot(eye_center[0], eye_center[1], 'o', color=color, 
                markersize=10, label=f'{"左" if i == 0 else "右"}眼球中心')
    
    # 绘制瞳孔中心
    for i, (pupil_center, color) in enumerate([
        (sample['pupil_center_2D'][0], 'orange'),
        (sample['pupil_center_2D'][1], 'cyan')
    ]):
        ax.plot(pupil_center[0], pupil_center[1], 's', color=color,
                markersize=8, label=f'{"左" if i == 0 else "右"}瞳孔中心')
    
    # 绘制虹膜轮廓
    for i, (iris_contour, color) in enumerate([
        (sample['iris_contour_2D'][0], 'red'),
        (sample['iris_contour_2D'][1], 'blue')
    ]):
        ax.plot(iris_contour[:, 0], iris_contour[:, 1], '-', 
                color=color, linewidth=2, alpha=0.7)
    
    ax.legend()
    ax.set_title('GazeGene 3D眼球标注可视化')
    ax.axis('off')
    
    plt.tight_layout()
    plt.savefig('gazegene_visualization.png', dpi=150)
    plt.show()


# 运行可视化
dataset = GazeGeneDataset("./GazeGene_normalized")
sample = dataset.load_sample(subject_id=44, camera_id=0, frame_id=100)
visualize_gaze_sample(sample)

---

六、实验结果与基准测试

6.1 与真实数据集对比

数据集	类型	样本数	跨域泛化性能
MPIIGaze	真实	213K	基线
ETH-XGaze	真实	1.5M	+12%
GazeCapture	真实	2.5M	+8%
GazeGene	合成	1.0M	+18%

6.2 高分辨率图像优势

分辨率	MPIIGaze	ETH-XGaze	GazeGene
640×480	3.5° 误差	3.2° 误差	2.8° 误差
1280×720	-	2.9° 误差	2.5° 误差
1920×1080	-	-	2.3° 误差

---

七、IMS/DMS 应用启示

7.1 视线估计模型训练

# 使用 GazeGene 训练视线估计模型
import torch
import torch.nn as nn
from torch.utils.data import DataLoader

class GazeEstimationModel(nn.Module):
    """基于 GazeGene 的视线估计模型"""
    
    def __init__(self, backbone='resnet50', pretrained=True):
        super().__init__()
        self.backbone = torch.hub.load(
            'pytorch/vision', backbone, pretrained=pretrained
        )
        self.gaze_regressor = nn.Sequential(
            nn.Linear(2048, 512),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(512, 3)  # 3D 视线向量
        )
    
    def forward(self, x):
        features = self.backbone(x)
        gaze_vector = self.gaze_regressor(features)
        return gaze_vector
    
    def forward_with_eye_features(self, x):
        """返回视线向量 + 眼球特征"""
        features = self.backbone(x)
        gaze_vector = self.gaze_regressor(features)
        
        # 额外输出眼球特征（用于下游任务）
        eye_features = features[:, :256]
        
        return gaze_vector, eye_features


# 训练代码
def train_gaze_model():
    dataset = GazeGeneDataset("./GazeGene_normalized")
    dataloader = DataLoader(dataset, batch_size=64, shuffle=True, num_workers=8)
    
    model = GazeEstimationModel().cuda()
    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
    criterion = nn.MSELoss()
    
    for epoch in range(50):
        for batch in dataloader:
            images = batch['image'].cuda()
            gt_gaze = batch['gaze_vector'].cuda()
            
            pred_gaze = model(images)
            loss = criterion(pred_gaze, gt_gaze)
            
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        
        print(f"Epoch {epoch}, Loss: {loss.item():.4f}")

7.2 分心检测应用

# 基于视线落点的分心检测
def detect_distraction(gaze_vector, head_pose, roi_zones):
    """
    检测驾驶员是否分心
    
    Args:
        gaze_vector: 3D视线向量
        head_pose: 头部姿态矩阵
        roi_zones: 感兴趣区域（道路前方、仪表盘、手机位置等）
    
    Returns:
        str: 'normal' | 'distraction_type'
    """
    # 计算视线落点
    gaze_origin = head_pose[:3, 3]  # 头部位置
    gaze_direction = gaze_vector / np.linalg.norm(gaze_vector)
    
    # 检查视线是否落在 ROI 区域
    for zone_name, zone_bbox in roi_zones.items():
        if _gaze_intersects_zone(gaze_origin, gaze_direction, zone_bbox):
            if zone_name == 'road_ahead':
                return 'normal'
            else:
                return f'distracted_{zone_name}'  # 如 'distracted_phone'
    
    return 'normal'


def _gaze_intersects_zone(origin, direction, bbox):
    """判断视线是否与区域相交"""
    # 简化实现：射线-AABB 相交检测
    t_min, t_max = 0, 100  # 检测距离范围
    
    for i in range(3):  # x, y, z
        if abs(direction[i]) < 1e-6:
            if origin[i] < bbox[0][i] or origin[i] > bbox[1][i]:
                return False
        else:
            t1 = (bbox[0][i] - origin[i]) / direction[i]
            t2 = (bbox[1][i] - origin[i]) / direction[i]
            t_min = max(t_min, min(t1, t2))
            t_max = min(t_max, max(t1, t2))
    
    return t_min <= t_max

---

八、限制与未来工作

8.1 当前限制

限制	描述	潜在解决方案
域迁移	合成→真实存在域差距	域适应、域泛化技术
眼镜遮挡	未涵盖戴眼镜情况	扩展资产库
极端光照	逆光、阴影场景有限	增强光照多样性

8.2 扩展方向

# 建议的扩展研究
future_work = [
    "结合 GazeGene + Cosmos 3 生成动态视线序列",
    "引入眼动时序特征（扫视、注视、眨眼）",
    "多模态融合：视线 + 头部姿态 + 面部表情",
    "跨数据集泛化：GazeGene → MPIIGaze → 真实座舱"
]

---

九、总结

GazeGene 数据集的三大突破：

维度	贡献
标注质量	首次提供完整3D眼球结构标注，零标注误差
数据规模	1M+ 高分辨率样本，56受试者，9视角
应用价值	支撑视线估计、分心检测、疲劳识别等多任务

对 IMS/DMS 开发的建议：

1. 优先采用 GazeGene 作为视线估计预训练数据
2. 利用3D眼球标注开发精细化疲劳检测算法
3. 结合真实数据进行域适应，提升跨域泛化

---

参考文献

1. Bao, Y., Wang, Z., & Lu, F. (2025). GazeGene: Large-scale Synthetic Gaze Dataset with 3D Eyeball Annotations. CVPR 2025.
2. Zhang, X., et al. (2015). Appearance-based gaze estimation in the wild. CVPR 2015 (MPIIGaze).
3. Kellnhofer, D., et al. (2019). Gaze360: Physically unconstrained gaze estimation in the wild. ICCV 2019.

---

发布时间：2026-08-01
关键词：GazeGene、视线估计、合成数据集、3D眼球标注、CVPR 2025、DMS、疲劳检测
代码可用性：数据集已开源，提供 Python 加载器和可视化工具