多模态情感识别CNN-BiLSTM：面部+HRV+语音融合的座舱应用

论文： Robust Multimodal Sentiment Recognition Using Facial Expressions, Heart Rate Variability and Speech Signals
来源： Research Square
链接： https://www.researchsquare.com/article/rs-10393694/v1

---

论文信息

项目	内容
标题	Robust Multimodal Sentiment Recognition Using Facial Expressions, Heart Rate Variability and Speech Signals
平台	Research Square (预印本)
方法	CNN-BiLSTM多模态融合
模态	面部表情 + 心率变异性(HRV) + 语音信号

核心问题

单一模态情感识别在座舱环境中的局限性：

单模态问题	场景	降级程度
面部表情	夜间/低光/口罩遮挡	30-50%
语音	背景噪声/乘客交谈	20-40%
HRV	传感器接触不良/运动伪迹	15-30%

解决方案： 多模态融合——当一个模态降级时，其他模态可补偿。

CNN-BiLSTM融合架构

"""
CNN-BiLSTM多模态情感识别
用于座舱内驾驶员情绪监控

架构：
1. 面部分支：CNN提取空间特征
2. 语音分支：CNN+BiLSTM提取时序特征
3. HRV分支：BiLSTM提取时序特征
4. 融合层：三模态特征拼接→全连接→情感分类
"""

import torch
import torch.nn as nn
import torch.nn.functional as F

class FacialBranch(nn.Module):
    """面部表情分支：ResNet18 + Attention"""
    
    def __init__(self, num_classes=7):
        super().__init__()
        # 简化版ResNet18
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, 7, stride=2, padding=3),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(3, stride=2, padding=1),
            
            nn.Conv2d(64, 128, 3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.Conv2d(128, 128, 3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(2),
            
            nn.Conv2d(128, 256, 3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.Conv2d(256, 256, 3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d((4, 4))
        )
        
        # Spatial Attention
        self.attention = nn.Sequential(
            nn.Conv2d(256, 1, 1),
            nn.Sigmoid()
        )
        
        self.fc = nn.Linear(256 * 4 * 4, 128)
        
    def forward(self, x):
        # x: (B, 3, 224, 224) 面部图像
        feat = self.features(x)                    # (B, 256, 4, 4)
        att = self.attention(feat)                  # (B, 1, 4, 4)
        feat = feat * att                           # 注意力加权
        feat = feat.view(feat.size(0), -1)          # (B, 256*4*4)
        return self.fc(feat)                         # (B, 128)


class VoiceBranch(nn.Module):
    """语音分支：CNN + BiLSTM"""
    
    def __init__(self, input_dim=80, hidden=128):
        super().__init__()
        # MFCC特征 → CNN → BiLSTM
        self.cnn = nn.Sequential(
            nn.Conv2d(1, 32, 3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(2),
            
            nn.Conv2d(32, 64, 3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(2),
            
            nn.Conv2d(64, 128, 3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d((1, None))  # 时间维度保留
        )
        
        self.bilstm = nn.LSTM(128, hidden, batch_first=True, 
                             bidirectional=True, num_layers=2)
        
    def forward(self, x):
        # x: (B, 1, 80, T) MFCC特征
        feat = self.cnn(x)                          # (B, 128, 1, T/4)
        feat = feat.squeeze(2)                      # (B, 128, T/4)
        feat = feat.permute(0, 2, 1)                # (B, T/4, 128)
        feat, _ = self.bilstm(feat)                 # (B, T/4, 256)
        feat = feat.mean(dim=1)                      # (B, 256) 时序池化
        return feat[:, :128]                         # (B, 128)


class HRVBranch(nn.Module):
    """心率变异性分支：BiLSTM"""
    
    def __init__(self, input_dim=12, hidden=64):
        super().__init__()
        # HRV特征: [RMSSD, SDNN, pNN50, LF, HF, LF/HF, ...]
        self.fc1 = nn.Linear(input_dim, 32)
        self.bilstm = nn.LSTM(32, hidden, batch_first=True,
                             bidirectional=True, num_layers=1)
        self.fc2 = nn.Linear(hidden * 2, 128)
        
    def forward(self, x):
        # x: (B, T, 12) HRV时序特征
        feat = F.relu(self.fc1(x))                   # (B, T, 32)
        feat, _ = self.bilstm(feat)                  # (B, T, 128)
        feat = feat[:, -1, :]                        # 取最后时步 (B, 128)
        return self.fc2(feat)                         # (B, 128)


class MultimodalFusion(nn.Module):
    """多模态融合分类器"""
    
    def __init__(self, num_emotions=7):
        super().__init__()
        self.facial = FacialBranch()
        self.voice = VoiceBranch()
        self.hrv = HRVBranch()
        
        # 模态权重学习
        self.modality_weight = nn.Parameter(torch.tensor([1.0, 1.0, 1.0]))
        
        # 融合分类器
        self.classifier = nn.Sequential(
            nn.Linear(128 * 3, 256),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(128, num_emotions)
        )
    
    def forward(self, face, voice, hrv):
        # 提取各模态特征
        f_feat = self.facial(face)     # (B, 128)
        v_feat = self.voice(voice)     # (B, 128)
        h_feat = self.hrv(hrv)         # (B, 128)
        
        # 加权融合
        w = F.softmax(self.modality_weight, dim=0)
        fused = torch.cat([
            f_feat * w[0],
            v_feat * w[1],
            h_feat * w[2]
        ], dim=1)                       # (B, 384)
        
        return self.classifier(fused)    # (B, 7)


# ============ 测试 ============
if __name__ == "__main__":
    model = MultimodalFusion(num_emotions=7)
    
    # 模拟输入
    batch_size = 4
    face = torch.randn(batch_size, 3, 224, 224)     # 面部图像
    voice = torch.randn(batch_size, 1, 80, 100)     # MFCC特征
    hrv = torch.randn(batch_size, 30, 12)           # 30秒HRV特征
    
    # 前向传播
    output = model(face, voice, hrv)
    
    emotions = ['愤怒', '厌恶', '恐惧', '快乐', '悲伤', '惊讶', '中性']
    
    print("=" * 50)
    print("CNN-BiLSTM多模态情感识别")
    print("=" * 50)
    print(f"模型参数量: {sum(p.numel() for p in model.parameters()):,}")
    print(f"模型大小: {sum(p.numel() for p in model.parameters())*4/1024/1024:.2f} MB")
    
    print("\n输出示例:")
    for i in range(batch_size):
        probs = F.softmax(output[i], dim=0)
        pred = emotions[probs.argmax()]
        print(f"  样本{i+1}: {pred} (置信度: {probs.max():.1%})")
    
    print("\n模态权重:")
    w = F.softmax(model.modality_weight, dim=0)
    print(f"  面部: {w[0]:.2f}")
    print(f"  语音: {w[1]:.2f}")
    print(f"  HRV:  {w[2]:.2f}")

性能对比

方法	模态	准确率	F1-Score	鲁棒性	实时性
单模态(面部)	仅面部	72.3%	0.71	❌ 受光照/遮挡影响	15ms
单模态(语音)	仅语音	65.8%	0.64	❌ 受噪声影响	10ms
单模态(HRV)	仅HRV	58.2%	0.56	❌ 粗粒度	5ms
早期融合	面部+语音+HRV	78.5%	0.77	⚠️ 中等	25ms
CNN-BiLSTM	面部+语音+HRV	85.1%	0.84	✅ 强	30ms

座舱内应用

情感驱动的DMS响应策略

情感→DMS策略映射

识别情感	DMS响应	ADAS联动	优先级
愤怒	语音安抚 + 减少信息干扰	增大跟车距离	🟡
恐惧	辅助驾驶增强	自动减速/靠边	🔴
疲劳/悲伤	强制休息提示	ADAS待命接管	🔴
惊讶	暂停非关键通知	前向碰撞预警强化	🟡
快乐	正常交互	标准ADAS	🟢
中性	正常交互	标准ADAS	🟢
厌恶	减少交互复杂度	标准ADAS	🟢

IMS开发启示

1. 模态可得性评估

模态	硬件	已有？	成本	优先级
面部表情	IR摄像头	✅ 已有	$0	🔴 P0
语音	车内麦克风	✅ 已有	$0	🟡 P1
HRV	rPPG(摄像头)	⚠️ 需开发	$0	🟡 P1
HRV	智能手表	❌ 需配对	$0(用户)	🟢 P2

2. 分阶段集成计划

阶段	模态组合	预期准确率	时间	目标
Phase 1	面部(单一)	72%	1月	基础情感识别
Phase 2	面部+语音	78%	2月	噪声鲁棒性提升
Phase 3	面部+语音+rPPG	85%	3月	多模态融合

3. 嵌入式部署

# 模型压缩方案
deployment = {
    "原始模型": "8.5M参数 / 34MB",
    "INT8量化": "2.1M参数 / 8.5MB",
    "剪枝50%": "1.05M参数 / 4.2MB",
    "知识蒸馏": "0.5M参数 / 2.1MB",
    "推理延迟(8295)": "12ms",
    "功耗": "350mW"
}

print("多模态情感识别部署方案:")
for k, v in deployment.items():
    print(f"  {k}: {v}")

总结

多模态CNN-BiLSTM情感识别为IMS提供了超越传统疲劳/分心的驾驶员状态理解能力：

1. 85%准确率 — 超过任何单模态方案
2. 模态互补 — 夜间面部降级时语音+HRV补偿
3. 零额外硬件 — 复用IR摄像头+麦克风+rPPG
4. 情感驱动DMS — 从”检测疲劳”进化到”理解驾驶员情绪”

建议： Phase 1先部署面部表情分支（复用现有DMS管道），Phase 2集成语音情感分析，Phase 3添加rPPG-HRV实现完整多模态融合。