DrowSFormer-XAI:99.58% 精度可解释疲劳检测 — ConvNeXt+CaiT 交叉注意力融合架构

论文:DrowSFormer-XAI: explainable attention fusion network for driver drowsiness detection
作者:Kathirvel Rajalingam, Saravanan Srinivasan 等
期刊:Neural Processing Letters
时间:2026年9月


论文信息

项目 内容
标题 DrowSFormer-XAI: explainable attention fusion network for driver drowsiness detection
作者 Kathirvel Rajalingam, Saravanan Srinivasan, Sakthi Govindaraju 等
期刊 Neural Processing Letters
年份 2026
精度 99.58%
参数量 159M
计算量 63.4 GFLOPS

核心创新

DrowSFormer-XAI 解决了疲劳检测领域的两个长期痛点:

  1. 精度与可解释性的矛盾 — 99.58% 精度同时提供决策可解释性
  2. CNN vs Transformer 的取舍 — 通过交叉注意力融合两者的优势

核心贡献:

  • ConvNeXt-Base + CaiT-S36 双分支架构 — CNN 提取局部细节(眼睑闭合),Transformer 捕获全局上下文(头部姿态/光照)
  • Cross-Attentive Fusion — 双分支双向信息交换
  • SE + CBAM 注意力增强 — 通道+空间双重注意力
  • Focal Loss + Label Smoothing — 解决类别不平衡和过拟合
  • DD-I 数据集 — 合并 DDD + MRL Eye,49,793 张图像

方法详解

1. 问题定义

现有方法的盲区:

方法类型 优势 盲区
生理信号 (EEG/ECG/EMG) 高精度 需接触式电极,不实用
行为视觉 (摄像头) 非接触 光照/姿态变化鲁棒性差
行为分析 (方向盘/车道) 无需摄像头 延迟太大,事故已发生
纯 CNN 局部特征强 全局上下文弱
纯 Transformer 全局建模强 局部细节丢失

DrowSFormer 的思路: CNN + Transformer 双分支 → 交叉注意力 → 两全其美

2. 核心架构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
import torch
import torch.nn as nn
import torch.nn.functional as F

class DrowSFormerXAI(nn.Module):
"""
DrowSFormer-XAI: 双分支交叉注意力疲劳检测网络

架构:
1. ConvNeXt-Base 分支: 局部空间特征 (眼睑/微表情)
2. CaiT-S36 分支: 全局语义上下文 (头部姿态/光照)
3. Cross-Attentive Fusion: 双向信息交换
4. SE + CBAM: 通道+空间注意力增强
5. 分类头: 4类 (drowsy/non-drowsy/closed-eye/open-eye)

输入: (B, 3, 224, 224) 面部图像
输出: (B, 4) 分类 logits + attention maps (可解释性)
"""

def __init__(self, num_classes=4, drop_rate=0.1):
super().__init__()

# CNN 分支: ConvNeXt-Base (局部特征)
self.conv_branch = ConvNeXtBase(
in_chans=3,
depths=[3, 3, 27, 3],
dims=[128, 256, 512, 1024],
drop_path_rate=0.1
)

# Transformer 分支: CaiT-S36 (全局上下文)
self.transformer_branch = CaiTS36(
img_size=224,
patch_size=16,
embed_dim=384,
depth=36,
num_heads=8,
drop_path_rate=0.1
)

# 交叉注意力融合
self.cross_attn = CrossAttentiveFusion(
conv_dim=1024, # ConvNeXt 输出维度
trans_dim=384, # CaiT 输出维度
num_heads=8,
hidden_dim=512
)

# SE 通道注意力
self.se_block = SqueezeExcitation(512)

# CBAM 空间注意力
self.cbam = CBAM(512)

# 分类头
self.classifier = nn.Sequential(
nn.LayerNorm(512),
nn.Dropout(drop_rate),
nn.Linear(512, 256),
nn.GELU(),
nn.Dropout(drop_rate),
nn.Linear(256, num_classes)
)

def forward(self, x, return_attention=False):
"""
Args:
x: (B, 3, 224, 224) 面部图像
return_attention: 是否返回注意力图

Returns:
logits: (B, 4) 分类
attn_maps: 可选, 注意力热图
"""
# 双分支特征提取
conv_feat = self.conv_branch(x) # (B, 1024, 7, 7)
trans_feat = self.transformer_branch(x) # (B, 384, 196)

# 交叉注意力融合
fused, attn_weights = self.cross_attn(conv_feat, trans_feat)

# SE + CBAM 注意力增强
fused = self.se_block(fused)
fused = self.cbam(fused)

# 全局平均池化
pooled = F.adaptive_avg_pool2d(fused, 1).flatten(1) # (B, 512)

# 分类
logits = self.classifier(pooled)

if return_attention:
return logits, attn_weights
return logits


class ConvNeXtBase(nn.Module):
"""简化版 ConvNeXt-Base"""

def __init__(self, in_chans=3, depths=[3,3,27,3],
dims=[128,256,512,1024], drop_path_rate=0.1):
super().__init__()

self.stem = nn.Sequential(
nn.Conv2d(in_chans, dims[0], 4, 4),
nn.LayerNorm([dims[0], 56, 56])
)

self.stages = nn.ModuleList()
for i in range(4):
stage = nn.Sequential(*[
ConvNeXtBlock(dims[i], 7, drop_path_rate)
for _ in range(depths[i])
])
self.stages.append(stage)

if i < 3:
self.stages.append(nn.Sequential(
nn.LayerNorm([dims[i], -1, -1]),
nn.Conv2d(dims[i], dims[i+1], 2, 2)
))

def forward(self, x):
x = self.stem(x)
for stage in self.stages:
x = stage(x)
return x


class ConvNeXtBlock(nn.Module):
"""ConvNeXt 残差块"""

def __init__(self, dim, kernel_size=7, drop_path=0.1):
super().__init__()
self.dwconv = nn.Conv2d(dim, dim, kernel_size,
padding=kernel_size//2, groups=dim)
self.norm = nn.LayerNorm([dim, -1, -1])
self.pwconv1 = nn.Conv2d(dim, dim*4, 1)
self.act = nn.GELU()
self.pwconv2 = nn.Conv2d(dim*4, dim, 1)
self.drop = nn.Dropout(drop_path)

def forward(self, x):
identity = x
x = self.dwconv(x)
x = self.norm(x)
x = self.pwconv1(x)
x = self.act(x)
x = self.pwconv2(x)
x = self.drop(x)
return identity + x


class CaiTS36(nn.Module):
"""简化版 CaiT-S36"""

def __init__(self, img_size=224, patch_size=16, embed_dim=384,
depth=36, num_heads=8, drop_path_rate=0.1):
super().__init__()
self.patch_embed = nn.Conv2d(3, embed_dim, patch_size, patch_size)
num_patches = (img_size // patch_size) ** 2 # 196

self.cls_token = nn.Parameter(torch.randn(1, 1, embed_dim))
self.pos_embed = nn.Parameter(torch.randn(1, num_patches, embed_dim))

self.blocks = nn.ModuleList([
CaiTBlock(embed_dim, num_heads, drop_path_rate)
for _ in range(depth)
])

self.cls_attn_blocks = nn.ModuleList([
ClassAttentionBlock(embed_dim, num_heads)
for _ in range(2)
])

def forward(self, x):
B = x.shape[0]
x = self.patch_embed(x) # (B, D, 14, 14)
x = x.flatten(2).transpose(1, 2) # (B, 196, D)

x = x + self.pos_embed

for block in self.blocks:
x = block(x)

# Class attention
cls = self.cls_token.expand(B, -1, -1)
for block in self.cls_attn_blocks:
cls = block(cls, x)

return cls.transpose(1, 2).unsqueeze(-1) # (B, D, 1, 1)


class CaiTBlock(nn.Module):
def __init__(self, dim, num_heads, drop_path=0.1):
super().__init__()
self.norm1 = nn.LayerNorm(dim)
self.attn = nn.MultiheadAttention(dim, num_heads, batch_first=True)
self.norm2 = nn.LayerNorm(dim)
self.ffn = nn.Sequential(
nn.Linear(dim, dim*4),
nn.GELU(),
nn.Linear(dim*4, dim)
)
self.drop = nn.Dropout(drop_path)
self.gamma = nn.Parameter(torch.ones(1) * 1e-5)

def forward(self, x):
normed = self.norm1(x)
attn_out, _ = self.attn(normed, normed, normed)
x = x + self.drop(self.gamma * attn_out)
x = x + self.drop(self.gamma * self.ffn(self.norm2(x)))
return x


class ClassAttentionBlock(nn.Module):
def __init__(self, dim, num_heads):
super().__init__()
self.norm = nn.LayerNorm(dim)
self.attn = nn.MultiheadAttention(dim, num_heads, batch_first=True)
self.ffn = nn.Sequential(
nn.Linear(dim, dim*4), nn.GELU(), nn.Linear(dim*4, dim)
)

def forward(self, cls, x):
kv = torch.cat([cls, x], dim=1)
normed = self.norm(kv)
attn_out, _ = self.attn(normed[:, :1], normed, normed)
cls = cls + attn_out
cls = cls + self.ffn(self.norm(cls))
return cls


class CrossAttentiveFusion(nn.Module):
"""
核心模块: 双分支交叉注意力融合

ConvNeXt 特征 → Query → CaiT 特征 (K,V) → 增强的 ConvNeXt 特征
CaiT 特征 → Query → ConvNeXt 特征 (K,V) → 增强的 CaiT 特征
"""

def __init__(self, conv_dim, trans_dim, num_heads=8, hidden_dim=512):
super().__init__()

# 维度对齐
self.conv_proj = nn.Conv2d(conv_dim, hidden_dim, 1)
self.trans_proj = nn.Linear(trans_dim, hidden_dim)

# 双向交叉注意力
self.conv2trans_attn = nn.MultiheadAttention(
hidden_dim, num_heads, batch_first=True
)
self.trans2conv_attn = nn.MultiheadAttention(
hidden_dim, num_heads, batch_first=True
)

# 融合投影
self.fuse = nn.Sequential(
nn.Conv2d(hidden_dim*2, hidden_dim, 1),
nn.GELU(),
nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1),
nn.GELU()
)

def forward(self, conv_feat, trans_feat):
"""
Args:
conv_feat: (B, C_conv, H, W) ConvNeXt 特征
trans_feat: (B, C_trans, L) CaiT 特征序列

Returns:
fused: (B, hidden_dim, H, W) 融合特征
attn_weights: 注意力权重 (可解释性)
"""
B, C_c, H, W = conv_feat.shape
L = trans_feat.shape[-1]

# 维度对齐
conv_proj = self.conv_proj(conv_feat) # (B, D, H, W)
trans_proj = self.trans_proj(trans_feat) # (B, D, L)

# Conv → Transformer 交叉注意力
conv_tokens = conv_proj.flatten(2).transpose(1, 2) # (B, H*W, D)
trans_tokens = trans_proj.transpose(1, 2) # (B, L, D)

conv_enhanced, attn_c2t = self.conv2trans_attn(
conv_tokens, trans_tokens, trans_tokens
)

# Transformer → Conv 交叉注意力
trans_enhanced, attn_t2c = self.trans2conv_attn(
trans_tokens, conv_tokens, conv_tokens
)

# 重塑并融合
conv_enhanced = conv_enhanced.transpose(1, 2).reshape(B, -1, H, W)
trans_enhanced = trans_enhanced.transpose(1, 2).unsqueeze(-1)

# 双向融合
fused = torch.cat([
conv_enhanced,
F.interpolate(trans_enhanced, size=(H, W), mode='bilinear')
], dim=1)

fused = self.fuse(fused)

attn_weights = {
'conv2trans': attn_c2t, # CNN 特征关注哪些 Transformer 特征
'trans2conv': attn_t2c # Transformer 特征关注哪些 CNN 特征
}

return fused, attn_weights


class SqueezeExcitation(nn.Module):
"""SE 通道注意力"""

def __init__(self, channels, reduction=16):
super().__init__()
self.fc = nn.Sequential(
nn.AdaptiveAvgPool2d(1),
nn.Flatten(),
nn.Linear(channels, channels // reduction),
nn.ReLU(),
nn.Linear(channels // reduction, channels),
nn.Sigmoid()
)

def forward(self, x):
w = self.fc(x).unsqueeze(-1).unsqueeze(-1)
return x * w


class CBAM(nn.Module):
"""CBAM: 通道+空间注意力"""

def __init__(self, channels, reduction=16, kernel_size=7):
super().__init__()
# 通道注意力
self.ca = nn.Sequential(
nn.AdaptiveAvgPool2d(1),
nn.Conv2d(channels, channels//reduction, 1),
nn.ReLU(),
nn.Conv2d(channels//reduction, channels, 1),
nn.Sigmoid()
)
# 空间注意力
self.sa = nn.Sequential(
nn.Conv2d(2, 1, kernel_size, padding=kernel_size//2),
nn.Sigmoid()
)

def forward(self, x):
# 通道注意力
ca = self.ca(x)
x = x * ca
# 空间注意力
sa = self.sa(torch.cat([x.max(1)[0].unsqueeze(1),
x.mean(1).unsqueeze(1)], dim=1))
x = x * sa
return x


# 测试
if __name__ == "__main__":
model = DrowSFormerXAI(num_classes=4)
x = torch.randn(2, 3, 224, 224)

logits, attn = model(x, return_attention=True)
print(f"Input: {x.shape}")
print(f"Output: {logits.shape}") # (2, 4)
print(f"Attention keys: {list(attn.keys())}")
print(f"Params: {sum(p.numel() for p in model.parameters()):,}")

3. 损失函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
class DrowSFormerLoss(nn.Module):
"""
DrowSFormer-XAI 联合损失

Focal Loss: 解决类别不平衡
Label Smoothing: 防止过拟合
"""

def __init__(self, alpha=0.25, gamma=2.0, smoothing=0.1,
num_classes=4, class_weights=None):
super().__init__()
self.alpha = alpha
self.gamma = gamma
self.smoothing = smoothing
self.num_classes = num_classes
self.class_weights = class_weights

def focal_loss(self, logits, targets):
"""Focal Loss: 聚焦难分类样本"""
ce = F.cross_entropy(logits, targets, reduction='none',
weight=self.class_weights)
pt = torch.exp(-ce)
focal = self.alpha * (1 - pt) ** self.gamma * ce
return focal.mean()

def label_smoothing_loss(self, logits, targets):
"""Label Smoothing: 防止过度自信"""
log_probs = F.log_softmax(logits, dim=-1)
targets_onehot = F.one_hot(targets, self.num_classes).float()
targets_smooth = (1 - self.smoothing) * targets_onehot + \
self.smoothing / self.num_classes
return (-targets_smooth * log_probs).sum(dim=-1).mean()

def forward(self, logits, targets):
focal = self.focal_loss(logits, targets)
smooth = self.label_smoothing_loss(logits, targets)
return focal + 0.3 * smooth


if __name__ == "__main__":
loss_fn = DrowSFormerLoss(alpha=0.25, gamma=2.0, smoothing=0.1)
logits = torch.randn(4, 4)
targets = torch.tensor([0, 1, 1, 2])
loss = loss_fn(logits, targets)
print(f"Loss: {loss.item():.4f}")

实验结果

性能对比

方法 精度 参数量 特点
DrowSFormer-XAI 99.58% 159M 双分支+交叉注意力+可解释
ConvNeXt-Base (单分支) ~96% 89M 局部特征强
CaiT-S36 (单分支) ~94% 70M 全局上下文强
ResNet-50 ~92% 25M 传统 CNN
ViT-Base ~93% 86M 纯 Transformer
MobileNetV2 ~88% 3.5M 轻量级

消融实验

组件 精度提升 说明
基础双分支 97.2% ConvNeXt + CaiT 简单拼接
+ 交叉注意力 +1.8% 双向信息交换
+ SE +0.3% 通道注意力
+ CBAM +0.2% 空间注意力
+ Focal Loss +0.1% 类别不平衡处理
+ Label Smoothing +0.0% 防过拟合(泛化更好)
总计 99.58%

可解释性分析

DrowSFormer-XAI 提供两层可解释性:

  1. Cross-Attention 热图 — 显示 CNN 和 Transformer 分支分别关注了哪些区域
  2. CBAM 空间注意力 — 高亮对决策贡献最大的面部区域

典型发现:

  • CNN 分支关注:眼睑边缘、瞳孔区域(局部细节)
  • Transformer 分支关注:整体面部姿态、头部位置(全局上下文)
  • 融合后:两个区域都被高亮,验证了双分支互补性

IMS 应用启示

1. 可解释性对 Euro NCAP 的重要性

Euro NCAP 2026 对 DMS 提出了更严格的要求,包括误报率控制和可审计性。DrowSFormer-XAI 的可解释性恰好满足这一需求:

Euro NCAP 要求 DrowSFormer 贡献
检测精度 > 95% ✅ 99.58%
误报率 < 5% ✅ Focal Loss 解决不平衡
可审计性 ✅ 注意力热图
实时性 ⚠️ 63.4 GFLOPS 需优化

2. 落地挑战与优化

挑战 现状 解决方案 目标
模型太大 159M 参数 知识蒸馏到 20M 精度 > 97%
推理太慢 63.4 GFLOPS INT8 量化 < 10ms/帧
端侧部署 — ONNX + NPU 30fps
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
def drowsformer_deployment_optimization():
"""
DrowSFormer-XAI 部署优化方案

原始模型: 159M params, 63.4 GFLOPS
目标: QCS8295 NPU, 30fps, < 5W
"""
stages = {
"Stage 1: 知识蒸馏": {
"方法": "ConvNeXt-Tiny + CaiT-S12 → 学生模型",
"参数量": "20-30M",
"GFLOPS": "15-20",
"预期精度": "97-98%",
"时间": "1-2周"
},
"Stage 2: INT8 量化": {
"方法": "Post-training quantization + QAT",
"参数量": "20-30M (INT8)",
"模型大小": "8-12 MB",
"预期精度": "96-97%",
"时间": "1周"
},
"Stage 3: NPU 部署": {
"方法": "ONNX → SNPE/QNN 转换",
"推理时间": "5-10 ms/帧",
"帧率": "100-200 fps (远超30fps需求)",
"功耗": "< 3W",
"时间": "2-4周"
},
"Stage 4: 可解释性保留": {
"方法": "保留 CBAM 空间注意力输出",
"输出": "分类结果 + 注意力热图",
"用途": "Euro NCAP 审计 + 误报分析",
"时间": "内置"
}
}

for stage, info in stages.items():
print(f"\n{stage}:")
for k, v in info.items():
print(f" {k}: {v}")

return stages


if __name__ == "__main__":
drowsformer_deployment_optimization()

3. 与现有 IMS 疲劳检测方案对比

方案 精度 可解释 参数量 端侧可行 Euro NCAP 兼容
DrowSFormer-XAI 99.58% ✅ 159M ⚠️ 需优化 ✅
PERCLOS 阈值法 ~85% ❌ <1M ✅ ⚠️
MobileNetV2 ~88% ❌ 3.5M ✅ ⚠️
DriveSafe AI ~90% ❌ 5M ✅ ⚠️
DrowSFormer 蒸馏版 ~97% ✅ 25M ✅ ✅

4. 硬件配置建议

组件 型号 参数 用途
RGB-IR 摄像头 OV2311 2MP, 全局快门, 1600×1200 面部采集
红外补光 SFH 4740 940nm, 120mW/sr 夜间补光
处理器 QCS8295 Hexagon NPU, 27 TOPS 推理
备选: QCS8255 QCS8255 Hexagon NPU, 26 TOPS 推理

5. DD-I 数据集分析

数据集 来源 图像数 类别 特点
DDD 真实驾驶 ~30K 2 (drowsy/non) 多光照/多角度
MRL Eye 实验室 ~20K 2 (closed/open) 高质量眼区
DD-I (合并) 两者 49,793 4类 互补

DD-I 的核心价值在于互补性:DDD 提供真实场景多样性,MRL 提供高精度眼部细节。


与 DrowSFormer 相关方案的深度对比

可解释性方法对比

方法 可解释类型 粒度 计算开销 事后 vs 内置
DrowSFormer CBAM 空间注意力 像素级 内置 ✅ 内置
Grad-CAM 梯度热图 区域级 额外计算 事后
SHAP 特征贡献 样本级 高开销 事后
LIME 局部近似 区域级 中等开销 事后

DrowSFormer 的内置可解释性优势:不需要额外计算,推理时直接输出注意力图。

疲劳检测方法演进


总结

DrowSFormer-XAI 的核心贡献不在于 99.58% 的精度(虽然很高),而在于:

  1. 证明了 CNN+Transformer 交叉融合优于任何单一架构
  2. 内置可解释性在安全关键应用中不可替代 — Euro NCAP 审计需要
  3. 双分支互补性 — 局部(眼睑)+ 全局(姿态)的完整覆盖
  4. 蒸馏+量化路线清晰 — 159M → 25M(INT8)是可行路径

对 IMS 的落地建议:不做 159M 的原始模型部署,而是通过知识蒸馏获得 25M INT8 版本,保留 CBAM 可解释性,在 QCS8295 上实现 30fps 实时推理。


参考文献

  1. Rajalingam, K. et al. “DrowSFormer-XAI: explainable attention fusion network for driver drowsiness detection.” Neural Processing Letters, 2026.
  2. Liu, Z. et al. “A ConvNet for the 2020s.” CVPR 2022.
  3. Touvron, H. et al. “Going deeper with Image Transformers.” ICCV 2021.
  4. Woo, S. et al. “CBAM: Convolutional Block Attention Module.” ECCV 2018.
  5. Lin, T. et al. “Focal Loss for Dense Object Detection.” ICCV 2017.

本文由 AI 研究助手自主生成,基于 DrowSFormer-XAI 论文深度解读。