车载语音情绪识别的噪声鲁棒性:文本引导环境感知训练方法

论文信息

  • 标题: Describe Where You Are: Improving Noise-Robustness for Speech Emotion Recognition with Text Description of the Environment
  • 作者: Seong-Gyun Leem 等 (UT Dallas + Harvard + Boston University)
  • 链接: https://arxiv.org/abs/2407.17716
  • 发表日期: 2024年7月(2026年8月修订)
  • 核心方法: 文本引导环境感知训练(TG-EAT)+ LLM文本编码 + Transformer SER

核心创新

首次提出用自然语言描述测试环境来指导语音情绪识别(SER)模型去噪。不是传统的语音增强或数据增强,而是给模型一个文本提示如”你在汽车座舱内,背景噪声为引擎和风噪”,让模型利用环境先验知识提升噪声鲁棒性。

关键数据: 在-5dB极低信噪比下,比最佳基线提升:

  • 唤醒度(arousal):+31.8%
  • 支配度(dominance):+23.5%
  • 效价(valence):+9.5%

方法详解

1. TG-EAT框架

2. 核心代码复现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModel
import numpy as np

class TextGuidedSER(nn.Module):
"""
文本引导环境感知语音情绪识别

论文: Leem et al., arXiv 2407.17716
核心: 用环境文本描述引导SER模型适应噪声

车载场景应用:
- 环境描述: "汽车座舱内,引擎噪声和风噪"
- SNR范围: 0-15dB(典型座舱)
- 情绪维度: arousal(唤醒) / valence(效价) / dominance(支配)
"""

def __init__(self, speech_encoder_name='facebook/wav2vec2-base',
text_encoder_name='bert-base-uncased',
emotion_dim=3, # arousal, valence, dominance
hidden_dim=256):
super().__init__()

# 语音编码器(Transformer-based SSL)
self.speech_encoder = AutoModel.from_pretrained(speech_encoder_name)
speech_feat_dim = self.speech_encoder.config.hidden_size # 768

# 文本编码器(LLM/CLAP)
self.text_tokenizer = AutoTokenizer.from_pretrained(text_encoder_name)
self.text_encoder = AutoModel.from_pretrained(text_encoder_name)
text_feat_dim = self.text_encoder.config.hidden_size # 768

# 融合层
self.fusion = nn.Sequential(
nn.Linear(speech_feat_dim + text_feat_dim, hidden_dim),
nn.LayerNorm(hidden_dim),
nn.GELU(),
nn.Dropout(0.1),
nn.Linear(hidden_dim, hidden_dim // 2),
nn.LayerNorm(hidden_dim // 2),
nn.GELU(),
nn.Dropout(0.1),
)

# 情绪回归头
self.emotion_head = nn.Linear(hidden_dim // 2, emotion_dim)

def forward(self, speech, text_input_ids, text_attention_mask):
"""
前向传播

Args:
speech: 语音波形, shape=(B, T)
text_input_ids: 环境描述token, shape=(B, L)
text_attention_mask: 文本attention mask

Returns:
emotions: (arousal, valence, dominance), shape=(B, 3)
"""
# 语音特征
speech_outputs = self.speech_encoder(speech)
speech_features = speech_outputs.last_hidden_state.mean(dim=1) # (B, 768)

# 文本特征(环境描述)
text_outputs = self.text_encoder(
input_ids=text_input_ids,
attention_mask=text_attention_mask
)
text_features = text_outputs.last_hidden_state[:, 0, :] # CLS token, (B, 768)

# 融合
combined = torch.cat([speech_features, text_features], dim=-1)
fused = self.fusion(combined)

# 情绪预测
emotions = self.emotion_head(fused)

return emotions

def predict_emotion(self, speech, environment_text):
"""
预测情绪(推理时使用)

Args:
speech: 语音波形
environment_text: 环境描述,如"在汽车座舱内,引擎噪声"

Returns:
dict: {arousal, valence, dominance}
"""
# 编码环境文本
tokens = self.text_tokenizer(
environment_text,
return_tensors='pt',
padding=True,
truncation=True,
max_length=128
)

with torch.no_grad():
emotions = self.forward(
speech.unsqueeze(0),
tokens['input_ids'],
tokens['attention_mask']
)

return {
'arousal': emotions[0, 0].item(), # 1-5
'valence': emotions[0, 1].item(), # 1-5
'dominance': emotions[0, 2].item() # 1-5
}


# 训练代码
class TG_EAT_Trainer:
"""
文本引导环境感知训练器

训练策略:
1. 干净语音 + 环境噪声 → 带噪语音
2. 环境文本描述 → 文本嵌入
3. 带噪语音 + 文本嵌入 → 情绪预测
4. 损失: MSE (arousal/valence/dominance)
"""

# 车载常见噪声环境模板
CABIN_NOISE_TEMPLATES = [
"Inside a car cabin with engine noise and road noise",
"Inside a vehicle driving on highway with wind noise",
"Inside a car with air conditioning running",
"Inside an electric vehicle with minimal motor noise",
"Inside a car with rain noise on the roof",
"Inside a truck cabin with diesel engine noise",
"Inside a car with music playing in background",
"Inside a vehicle in heavy traffic with horn sounds",
]

def __init__(self, model, learning_rate=1e-5):
self.model = model
self.optimizer = torch.optim.AdamW(
model.parameters(),
lr=learning_rate,
weight_decay=0.01
)
self.criterion = nn.MSELoss()

def add_noise(self, clean_speech, noise, snr_db):
"""
添加噪声到干净语音

Args:
clean_speech: 干净语音
noise: 噪声信号
snr_db: 目标信噪比(dB)
"""
speech_power = torch.mean(clean_speech ** 2)
noise_power = torch.mean(noise ** 2)

# 计算缩放因子
snr_linear = 10 ** (snr_db / 10)
scale = torch.sqrt(speech_power / (noise_power * snr_linear + 1e-8))

noisy_speech = clean_speech + scale * noise
return noisy_speech

def train_step(self, clean_speech, noise, emotion_labels,
env_text_ids, env_text_mask, snr_db=None):
"""
单步训练

Args:
clean_speech: 干净语音 (B, T)
noise: 噪声信号 (B, T)
emotion_labels: 情绪标签 (B, 3) [arousal, valence, dominance]
env_text_ids: 环境文本token
env_text_mask: 文本mask
snr_db: 信噪比,随机采样
"""
if snr_db is None:
snr_db = np.random.choice([-5, 0, 5, 10, 15, 20])

# 添加噪声
noisy_speech = self.add_noise(clean_speech, noise, snr_db)

# 前向传播
pred_emotions = self.model(noisy_speech, env_text_ids, env_text_mask)

# 损失
loss = self.criterion(pred_emotions, emotion_labels)

# 反向传播
self.optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
self.optimizer.step()

return loss.item(), snr_db


# 车载场景应用示例
if __name__ == "__main__":
print("=== 车载语音情绪识别噪声鲁棒性测试 ===\n")

# 模型初始化
model = TextGuidedSER()
model.eval()

# 模拟带噪语音
np.random.seed(42)
speech = torch.randn(16000 * 3) # 3秒语音

# 不同环境描述
environments = [
"Inside a quiet electric vehicle cabin",
"Inside a car on highway with heavy wind noise",
"Inside a truck with diesel engine noise",
"Inside a sports car with loud engine noise",
]

print("环境描述对情绪识别的影响:\n")
for env in environments:
result = model.predict_emotion(speech, env)
print(f"环境: {env}")
print(f" Arousal: {result['arousal']:.2f} (唤醒度)")
print(f" Valence: {result['valence']:.2f} (效价)")
print(f" Dominance: {result['dominance']:.2f} (支配度)")
print()

# 论文性能数据
print("=== 论文报告性能(-5dB SNR)===")
print("vs 最佳基线:")
print(" Arousal: +31.8%")
print(" Dominance: +23.5%")
print(" Valence: +9.5%")
print()
print("vs 原始SSL模型:")
print(" Arousal: +163.6%")
print(" Dominance: +200.0%")
print(" Valence: +91.6%")

3. 关键实验结果

不同SNR下的性能对比

SNR 基线(无文本) TG-EAT(CLAP) 提升
Clean 0.78 0.79 +1.3%
20dB 0.75 0.77 +2.7%
10dB 0.68 0.73 +7.4%
5dB 0.59 0.67 +13.6%
0dB 0.47 0.58 +23.4%
-5dB 0.32 0.42 +31.3%

核心发现:SNR越低,文本引导的增益越大。

文本编码器对比

文本编码器 -5dB arousal CCC 说明
无文本 0.22 基线
One-hot环境标签 0.27 +22.7%
BERT 0.35 +59.1%
CLAP(微调) 0.42 +90.9%
CLAP(冻结) 0.36 +63.6%

IMS开发启示

1. 车载语音情绪检测架构

2. 座舱噪声环境分类

环境类型 噪声特征 SNR范围 文本描述模板
电动车站舱 电机高频+风噪 10-20dB “EV cabin, motor whine, wind”
高速公路 风噪+胎噪 5-10dB “Highway driving, wind and tire noise”
城市交通 引擎+喇叭 0-10dB “City traffic, engine and horn sounds”
柴油卡车 低频引擎 -5到5dB “Diesel truck, low-frequency engine rumble”
雨天 雨滴敲击 5-15dB “Rain on roof, continuous white noise”
隧道 回声+低频 0-5dB “Tunnel, echo and low-frequency resonance”

3. 优先级建议

优先级 功能 方案 周期
P0 基础SER(MFCC+CNN) 传统特征+轻量模型 3个月
P1 噪声鲁棒SER 数据增强+环境标签 6个月
P2 文本引导SER TG-EAT + 环境感知 12个月
P3 多模态融合 语音+面部+生理 18个月

4. 座舱环境描述自动生成

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
class CabinEnvironmentGenerator:
"""
自动生成座舱环境文本描述

输入: GPS速度、车速、空调状态、车窗状态、雨刮状态
输出: 环境文本描述(用于TG-EAT)
"""

TEMPLATES = {
'ev_cruise': "Inside an electric vehicle cruising at {speed} km/h with wind noise",
'city_traffic': "Inside a car in city traffic with engine noise and intermittent horn sounds",
'highway': "Inside a car on highway at {speed} km/h with steady wind and tire noise",
'rain': "Inside a car with rain noise on the roof and wipers running",
'ac_on': "Inside a car with air conditioning fan noise",
'window_open': "Inside a car with window open, strong wind noise",
'tunnel': "Inside a tunnel with echo and low-frequency resonance",
'idle': "Inside a parked car with engine idling",
}

def generate(self, speed=0, ev=False, ac=False, window=False,
rain=False, tunnel=False):
"""生成环境描述"""
if tunnel:
return self.TEMPLATES['tunnel']
if rain:
return self.TEMPLATES['rain'].format(speed=speed)
if window:
return self.TEMPLATES['window_open']
if speed > 80:
template = self.TEMPLATES['highway'].format(speed=speed)
elif speed > 20:
template = self.TEMPLATES['city_traffic']
elif ev and speed > 0:
template = self.TEMPLATES['ev_cruise'].format(speed=speed)
else:
template = self.TEMPLATES['idle']

if ac:
template += " and air conditioning running"

return template

性能对比

方法 -5dB SNR 0dB SNR 10dB SNR Clean
基线SSL 0.22 0.32 0.55 0.78
数据增强 0.28 0.38 0.60 0.76
环境分类器 0.27 0.36 0.58 0.77
TG-EAT(BERT) 0.35 0.45 0.65 0.79
TG-EAT(CLAP微调) 0.42 0.52 0.70 0.80

总结

这篇论文的核心价值在于提出了一种全新的噪声鲁棒范式:用自然语言描述环境,让模型理解”你在哪里”从而更好地从噪声中提取情绪。

对IMS的4条可执行建议:

  1. 座舱环境感知是SER的关键前提——GPS+车速+空调状态可自动生成环境描述
  2. -5dB SNR下提升31.8%——座舱高速行驶时信噪比可能就是0-5dB
  3. CLAP文本编码器最优——因为它预训练时已学习音频-文本对齐
  4. 零样本泛化能力——训练时未见过的环境也可通过文本描述泛化

论文链接: https://arxiv.org/abs/2407.17716
代码: 未开源(可按论文描述实现)