视线解码超越分类:ECCV 2026眼动→自然语言描述

论文信息

项目 内容
标题 Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention
会议 ECCV 2026 (Poster)
链接 ECCV Poster Page
关键词 gaze-to-text, natural language, attention decoding, visual tasks

核心创新

将眼动视线解码为自然语言描述,而非传统的预定义类别分类。

范式转变

方面 传统方案 本文方案
输出格式 预定义类别标签 自然语言描述
任务定义 判别式分类 生成式描述
信息量 单一类别 丰富语义
适用场景 固定任务集 开放场景
模型架构 分类器 文本生成器

示例对比

场景 传统分类输出 gaze-to-text输出
看仪表盘 “dashboard” “驾驶员注视速度表,可能检查车速”
看手机 “phone” “驾驶员视线移向手机屏幕,疑似阅读消息”
看后视镜 “mirror” “驾驶员频繁查看内后视镜,关注后方交通”
看副驾 “passenger” “驾驶员与副驾乘客进行眼神交流”
看导航 “screen” “驾驶员查看中控导航屏幕路线指示”

方法概述

核心代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
import torch
import torch.nn as nn

class GazeToTextModel(nn.Module):
"""
视线→自然语言描述生成模型

范式:从gaze轨迹生成描述性文本
输入:gaze轨迹序列 + 视觉上下文
输出:自然语言描述
"""

def __init__(self, gaze_dim=4, d_model=256, vocab_size=10000,
max_seq_len=50, n_heads=8, n_layers=4):
super().__init__()

# Gaze轨迹编码
self.gaze_encoder = nn.Sequential(
nn.Linear(gaze_dim, d_model),
nn.LayerNorm(d_model),
nn.GELU(),
nn.Dropout(0.1)
)

# 位置编码
self.pos_encoding = nn.Parameter(
torch.randn(1, max_seq_len, d_model) * 0.02
)

# 视觉上下文编码(来自场景理解)
self.visual_context_encoder = nn.Linear(512, d_model) # 假设512维场景特征

# Transformer解码器(生成文本)
decoder_layer = nn.TransformerDecoderLayer(
d_model=d_model,
nhead=n_heads,
dim_feedforward=d_model * 4,
dropout=0.1,
batch_first=True
)
self.decoder = nn.TransformerDecoder(
decoder_layer, num_layers=n_layers
)

# 语言模型头
self.lm_head = nn.Linear(d_model, vocab_size)

def forward(self, gaze_seq, visual_context, target_tokens=None,
teacher_forcing_ratio=0.8):
"""
Args:
gaze_seq: (B, T_gaze, 4) gaze轨迹 [x, y, timestamp, velocity]
visual_context: (B, 512) 场景理解特征
target_tokens: (B, T_text) 目标文本token (训练时)
teacher_forcing_ratio: teacher forcing比例

Returns:
logits: (B, T_text, vocab_size)
"""
B = gaze_seq.size(0)

# 编码gaze
gaze_emb = self.gaze_encoder(gaze_seq) # (B, T_gaze, d)
gaze_emb = gaze_emb + self.pos_encoding[:, :gaze_seq.size(1)]

# 编码视觉上下文
ctx_emb = self.visual_context_encoder(visual_context) # (B, d)
ctx_emb = ctx_emb.unsqueeze(1) # (B, 1, d)

# 记忆 = gaze编码 + 视觉上下文
memory = torch.cat([ctx_emb, gaze_emb], dim=1) # (B, 1+T_gaze, d)

# 文本生成
if target_tokens is not None:
# 训练模式:teacher forcing
T_text = target_tokens.size(1)
tgt_emb = nn.functional.embedding(
target_tokens,
torch.randn(self.lm_head.out_features,
self.lm_head.in_features,
device=target_tokens.device)
)
tgt_emb = tgt_emb + self.pos_encoding[:, :T_text]

# 因果掩码
causal_mask = torch.triu(
torch.ones(T_text, T_text, device=target_tokens.device)
* float('-inf'), diagonal=1
)

decoded = self.decoder(
tgt_emb, memory,
tgt_mask=causal_mask
)
logits = self.lm_head(decoded)
return logits
else:
# 推理模式:自回归生成
generated = torch.ones(B, 1, dtype=torch.long,
device=gaze_seq.device) # <BOS>

for _ in range(50): # 最多生成50个token
tgt_emb = nn.functional.embedding(
generated,
torch.randn(self.lm_head.out_features,
self.lm_head.in_features,
device=generated.device)
)
tgt_emb = tgt_emb + self.pos_encoding[:, :generated.size(1)]

decoded = self.decoder(tgt_emb, memory)
next_logits = self.lm_head(decoded[:, -1])
next_token = next_logits.argmax(dim=-1, keepdim=True)
generated = torch.cat([generated, next_token], dim=1)

if (next_token == 2).all(): # <EOS>
break

return generated


# === DMS 集成方案 ===
class GazeDescriptionDMS:
"""将gaze-to-text集成到DMS的方案"""

# 描述模板映射(用于DMS决策)
DESCRIPTION_PATTERNS = {
'phone_check': r'phone|mobile|screen.*hand',
'fatigue': r'drowsy|tired|heavy.*eyes|slow.*blink',
'distraction': r'away.*road|off.*road|looking.*side',
'mirror_check': r'mirror|rear|behind',
'dashboard': r'speed|odometer|fuel|dashboard',
'navigation': r'nav|route|direction|GPS',
'passenger_interaction': r'passenger|side.*seat',
}

def gaze_to_dms_state(self, description):
"""将自然语言描述转为DMS状态"""
import re
states = []
for state, pattern in self.DESCRIPTION_PATTERNS.items():
if re.search(pattern, description, re.IGNORECASE):
states.append(state)

if not states:
return 'normal'
if 'phone_check' in states:
return 'phone_use_warning'
if 'fatigue' in states:
return 'fatigue_warning'
if 'distraction' in states:
return 'distraction_warning'
if len(states) >= 2:
return 'multi_task_warning'
return states[0]


# 测试
if __name__ == "__main__":
model = GazeToTextModel(gaze_dim=4, d_model=128, vocab_size=5000)

B = 2
T_gaze = 30 # 1秒@30fps
gaze = torch.randn(B, T_gaze, 4)
ctx = torch.randn(B, 512)

# 模拟文本目标
target = torch.randint(0, 5000, (B, 20))

logits = model(gaze, ctx, target_tokens=target)
print(f"Gaze输入: {gaze.shape}")
print(f"视觉上下文: {ctx.shape}")
print(f"文本logits: {logits.shape}")
print(f"参数量: {sum(p.numel() for p in model.parameters())/1e6:.2f}M")

# DMS集成示例
dms = GazeDescriptionDMS()
test_descs = [
"Driver looks at phone screen, possibly reading message",
"Driver eyes heavy, slow blinks detected",
"Driver checks rear mirror frequently",
"Driver gazes at navigation screen",
]
print("\n=== Gaze描述→DMS状态 ===")
for desc in test_descs:
state = dms.gaze_to_dms_state(desc)
print(f" '{desc}' → {state}")

IMS 开发启示

1. 从分类到描述的价值

传统分类 gaze-to-text 优势
“phone” “驾驶员右手持手机阅读消息,持续3秒” 更精确的上下文
“drowsy” “驾驶员眼睑下垂,PERCLOS 35%,疑似中度疲劳” 包含量化指标
“distracted” “驾驶员视线持续偏右,可能与副驾交谈” 行为原因推断

2. DMS 审计日志

gaze-to-text 可生成人类可读的DMS审计日志:

1
2
3
4
5
6
[14:23:01] Driver focused on road ahead
[14:23:05] Driver glanced at speedometer briefly
[14:23:12] Driver looked at phone for 2.3s (phone_use detected)
[14:23:15] Driver returned gaze to road, alert
[14:23:30] Driver eyes showing fatigue signs (PERCLOS 28%)
[14:23:45] Driver checking mirror, normal behavior

3. Euro NCAP 审核支持

Euro NCAP 要求 gaze-to-text 的价值
检测场景说明 自动生成行为描述
误报分析 可追溯每次决策依据
驾驶员状态记录 人类可读日志
系统验证 审核方可直接理解

局限性

  1. 生成质量 — 描述质量依赖训练数据和语言模型
  2. 延迟 — 文本生成比分类慢
  3. 一致性 — 同一行为可能生成不同描述
  4. 量化指标缺失 — 自然语言不如数值精确
  5. 训练数据 — 需要gaze-文本对标注,成本高

总结

gaze-to-text 代表了 DMS 从”分类”到”理解”的范式转变。对 IMS 团队:

  1. 目前不急于落地 — 技术尚早期,推理延迟待优化
  2. 审计日志价值大 — 可用LLM事后生成描述
  3. 与传统分类互补 — 分类做实时决策,描述做事后分析
  4. 未来方向 — 多模态LLM+gaze可能直接输出DMS决策
  5. 训练数据是新瓶颈 — 需要gaze-文本对标注

参考链接: