CNN-ViT多特征融合疲劳检测：DenseNet+Transformer混合架构解析

来源：ScienceDirect (June 2025)
链接：https://www.sciencedirect.com/science/article/pii/S2590005625000529

核心创新

首次融合CNN局部特征与ViT全局语义进行疲劳检测：

• 准确率：97.3%（单模型最高）
• 多特征融合：眼动+嘴部+头部姿态
• 实时性：实时推理（25FPS）

技术架构

混合架构设计

graph TB
    A[输入帧] --> B[人脸检测]
    B --> C1[眼部ROI]
    B --> C2[嘴部ROI]
    B --> C3[头部姿态]
    
    C1 --> D1[DenseNet121]
    C2 --> D2[VGG16]
    C3 --> D3[ResNet50]
    
    D1 --> E[特征拼接]
    D2 --> E
    D3 --> E
    
    E --> F[Vision Transformer]
    F --> G[分类头]
    G --> H[疲劳状态]

核心模块实现

import torch
import torch.nn as nn
import torchvision.models as models

class CNN_ViT_Drowsiness(nn.Module):
    """
    CNN-ViT混合疲劳检测模型
    
    架构：
    1. 多CNN分支提取局部特征
    2. ViT融合全局语义
    3. 多任务学习
    """
    
    def __init__(self, num_classes=2, feature_dim=768):
        super().__init__()
        
        # 眼部特征提取器（DenseNet121）
        self.eye_encoder = models.densenet121(pretrained=True)
        self.eye_encoder.classifier = nn.Identity()
        eye_dim = 1024
        
        # 嘴部特征提取器（VGG16）
        self.mouth_encoder = models.vgg16(pretrained=True)
        self.mouth_encoder.classifier = nn.Sequential(*list(self.mouth_encoder.classifier.children())[:-1])
        mouth_dim = 4096
        
        # 头部姿态提取器（ResNet50）
        self.head_encoder = models.resnet50(pretrained=True)
        self.head_encoder.fc = nn.Identity()
        head_dim = 2048
        
        # 特征投影层
        self.eye_proj = nn.Linear(eye_dim, feature_dim)
        self.mouth_proj = nn.Linear(mouth_dim, feature_dim)
        self.head_proj = nn.Linear(head_dim, feature_dim)
        
        # Vision Transformer
        self.vit = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(
                d_model=feature_dim,
                nhead=12,
                dim_feedforward=feature_dim * 4,
                dropout=0.1,
                batch_first=True
            ),
            num_layers=6
        )
        
        # 分类头
        self.classifier = nn.Sequential(
            nn.Linear(feature_dim, 256),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(256, num_classes)
        )
        
    def forward(self, eye_img, mouth_img, head_img):
        """
        Args:
            eye_img: (B, 3, 64, 64) 眼部图像
            mouth_img: (B, 3, 64, 64) 嘴部图像
            head_img: (B, 3, 224, 224) 头部图像
        
        Returns:
            logits: (B, num_classes)
        """
        # 提取各部位特征
        eye_feat = self.eye_encoder(eye_img)        # (B, 1024)
        mouth_feat = self.mouth_encoder(mouth_img)  # (B, 4096)
        head_feat = self.head_encoder(head_img)     # (B, 2048)
        
        # 投影到统一维度
        eye_feat = self.eye_proj(eye_feat)    # (B, 768)
        mouth_feat = self.mouth_proj(mouth_feat)  # (B, 768)
        head_feat = self.head_proj(head_feat)     # (B, 768)
        
        # 拼接为序列
        sequence = torch.stack([eye_feat, mouth_feat, head_feat], dim=1)  # (B, 3, 768)
        
        # Transformer融合
        fused = self.vit(sequence)  # (B, 3, 768)
        
        # 池化
        pooled = fused.mean(dim=1)  # (B, 768)
        
        # 分类
        return self.classifier(pooled)

多特征提取

class MultiFeatureExtractor:
    """
    多特征提取器
    
    提取：
    1. 眼部：PERCLOS、眨眼频率
    2. 嘴部：打哈欠频率
    3. 头部：点头/摇头频率
    """
    
    def __init__(self):
        self.face_detector = self._load_mtcnn()
        
    def extract_features(self, frame):
        """
        从单帧提取多部位图像
        
        Returns:
            eye_img: (3, 64, 64)
            mouth_img: (3, 64, 64)
            head_img: (3, 224, 224)
        """
        # 检测人脸
        boxes, landmarks = self.face_detector.detect(frame)
        
        if boxes is None:
            return None, None, None
        
        # 提取关键点
        left_eye = landmarks[0]
        right_eye = landmarks[1]
        mouth = landmarks[3]
        
        # 裁剪各部位
        eye_img = self._crop_eyes(frame, left_eye, right_eye)
        mouth_img = self._crop_mouth(frame, mouth)
        head_img = self._crop_head(frame, boxes[0])
        
        return eye_img, mouth_img, head_img
    
    def _crop_eyes(self, frame, left_eye, right_eye):
        """裁剪双眼区域"""
        # 计算眼部边界框
        x1 = int(min(left_eye[0], right_eye[0]) - 20)
        y1 = int(min(left_eye[1], right_eye[1]) - 15)
        x2 = int(max(left_eye[0], right_eye[0]) + 20)
        y2 = int(max(left_eye[1], right_eye[1]) + 15)
        
        # 裁剪并resize
        crop = frame[y1:y2, x1:x2]
        crop = cv2.resize(crop, (64, 32))  # 双眼区域较宽
        
        # 填充到64x64
        padded = np.zeros((64, 64, 3), dtype=np.uint8)
        padded[16:48, :, :] = crop
        
        return padded
    
    def _crop_mouth(self, frame, mouth_center):
        """裁剪嘴部区域"""
        x, y = mouth_center
        crop = frame[int(y-20):int(y+20), int(x-30):int(x+30)]
        crop = cv2.resize(crop, (64, 64))
        return crop

实验结果

各CNN骨干网络对比

骨干网络	参数量	Eye准确率	Mouth准确率	Head准确率
DenseNet121	8M	94.5%	-	-
VGG16	138M	-	92.3%	-
ResNet50	25M	-	-	93.8%
融合	171M	97.3%	96.1%	96.5%

消融实验

配置	准确率	说明
仅眼部特征	94.5%	单模态基线
仅嘴部特征	92.3%	单模态基线
仅头部特征	93.8%	单模态基线
眼部+嘴部	96.1%	双模态融合
眼部+头部	96.5%	双模态融合
全部融合+ViT	97.3%	最佳性能

跨数据集泛化

训练数据集	测试数据集	准确率
DDD	DDD	97.3%
DDD	NTHU-DDD	91.5%
混合数据集	混合数据集	95.8%

IMS应用启示

1. 模型优化策略

def optimize_for_deployment():
    """
    部署优化策略
    
    问题：171M参数太大
    方案：
    1. 知识蒸馏到单一网络
    2. 模型剪枝
    3. 量化
    """
    
    # 方案1：蒸馏到MobileNetV3
    student = MobileNetV3_Large(num_classes=2)
    teacher = CNN_ViT_Drowsiness()
    
    # 训练student模仿teacher
    distill_loss = nn.KLDivLoss()
    
    # 方案2：剪枝
    from torch.nn.utils import prune
    for name, module in teacher.named_modules():
        if isinstance(module, nn.Linear):
            prune.l1_unstructured(module, name='weight', amount=0.3)
    
    # 方案3：量化
    quantized = torch.quantization.quantize_dynamic(
        teacher, {nn.Linear}, dtype=torch.qint8
    )

2. 实时性优化

class RealTimePipeline:
    """
    实时疲劳检测管道
    
    优化点：
    1. 人脸检测跳帧
    2. 特征缓存复用
    3. 异步处理
    """
    
    def __init__(self):
        self.model = CNN_ViT_Drowsiness()
        self.feature_extractor = MultiFeatureExtractor()
        
        self.face_cache = None
        self.cache_lifetime = 5  # 帧
        
    def process_frame(self, frame):
        """处理单帧"""
        # 每N帧重新检测人脸
        if self.frame_count % self.cache_lifetime == 0:
            self.face_cache = self.feature_extractor.extract_features(frame)
        
        # 使用缓存的人脸位置
        eye_img, mouth_img, head_img = self.face_cache
        
        # 模型推断
        with torch.no_grad():
            logits = self.model(eye_img, mouth_img, head_img)
        
        return logits

3. Euro NCAP合规

要求	本方案	状态
多特征融合	眼+嘴+头	✅
实时性	25 FPS	✅
准确率>95%	97.3%	✅
夜间可用	需红外摄像头	⚠️

参考文献

1. CNN-ViT: A multi-feature learning based approach for driver drowsiness detection. ScienceDirect (2025).
2. Huang, G. et al. (2018). Densely Connected Convolutional Networks. CVPR.

---

总结： CNN-ViT多特征融合在精度上达到新高，但参数量较大需要蒸馏优化后部署。建议先验证精度收益是否值得计算开销。