rPPG 远程光电容积描记:从图神经网络到超短视频心率估计——驾驶员非接触心率监测新前沿

心率是疲劳监测的核心生理指标,但传统rPPG需要10秒视频窗口——太慢了。Reperio-rPPG用图神经网络建模心率周期性,Ultra-Short rPPG从2秒视频估计心率。两项突破直接赋能IMS实时疲劳监测。

论文一:Reperio-rPPG

项目 内容
标题 Reperio-rPPG: Relational Temporal Graph Neural Networks for Periodicity Learning in Remote Physiological Measurement
arXiv 2511.05946
代码 GitHub: deconasser/Reperio-rPPG
机构 越南国立大学
数据集 PURE, UBFC-rPPG, MMPD
创新 Swin Transformer + R-GCN + Graph Transformer

论文二:Ultra-Short rPPG

项目 内容
标题 Ultra-Short rPPG Estimation via Periodicity Guidance and Signal Reconstruction
期刊 Pattern Recognition
arXiv 2506.22078
机构 福建师范大学 / 台湾清华大学 / 世宗大学
创新 2秒视频估计心率 + 周期性引导 + 信号重建

核心创新对比

维度 Reperio-rPPG Ultra-Short rPPG
核心方法 图神经网络建模周期性 2秒超短期估计
视频窗口 10秒标准 2秒超短
架构 Swin + R-GCN + Graph Transformer rPPG模型 + 生成器重建
关键创新 图边关系编码帧间相位 周期一致性损失 + 频谱泄漏修复
数据增强 Temporal CutMix + NDF + MPOS 信号重建增强
优势场景 运动鲁棒 实时快速

方法详解

Reperio-rPPG 架构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
"""
Reperio-rPPG: 图神经网络驱动的rPPG估计

架构:Swin Transformer → R-GCN → Graph Transformer → rPPG信号
关键创新:将视频帧视为图节点,用边关系编码周期性
"""

import torch
import torch.nn as nn
from typing import Tuple

class ReperioRPPG(nn.Module):
"""
Reperio-rPPG 完整模型

1. Swin Transformer: 提取空间特征
2. R-GCN: 建模帧间关系(图边)
3. Graph Transformer: 捕获周期性和时序依赖
"""

def __init__(
self,
img_size: int = 128,
patch_size: int = 4,
in_chans: int = 3,
embed_dim: int = 96,
depths: list = [2, 2, 6, 2],
num_heads: list = [3, 6, 12, 24],
num_frames: int = 300, # 10秒 @ 30fps
num_relations: int = 4,
graph_hidden: int = 256
):
super().__init__()

# 1. Swin Transformer 空间特征提取
self.swin_encoder = SwinTransformer(
img_size=img_size,
patch_size=patch_size,
in_chans=in_chans,
embed_dim=embed_dim,
depths=depths,
num_heads=num_heads
)

# 空间池化 → 每帧一个特征向量
self.spatial_pool = nn.AdaptiveAvgPool2d(1)

# 2. R-GCN: 关系图卷积
self.rgcn = RelationalGCN(
in_dim=embed_dim,
hidden_dim=graph_hidden,
out_dim=graph_hidden,
num_relations=num_relations,
num_layers=3
)

# 3. Graph Transformer
self.graph_transformer = GraphTransformer(
d_model=graph_hidden,
nhead=8,
num_layers=4,
dim_feedforward=graph_hidden * 4
)

# 输出头
self.rppg_head = nn.Sequential(
nn.Linear(graph_hidden, graph_hidden // 2),
nn.GELU(),
nn.Linear(graph_hidden // 2, 1) # 每帧rPPG值
)

def forward(
self,
video: torch.Tensor # (B, T, C, H, W)
) -> torch.Tensor:
"""
Returns:
rppg_signal: (B, T) 估计的rPPG信号
"""
B, T, C, H, W = video.shape

# 1. 空间特征提取
video_flat = video.view(B * T, C, H, W)
spatial_features = self.swin_encoder(video_flat) # (B*T, D, H', W')
spatial_features = self.spatial_pool(spatial_features).squeeze(-1).squeeze(-1)
spatial_features = spatial_features.view(B, T, -1) # (B, T, D)

# 2. 构建图:帧为节点,关系为边
# 关系类型:邻近(±1帧)、周期(相同相位)、趋势(单调)、长程
graph_data = self._build_graph(spatial_features)

# 3. R-GCN 更新
graph_features = self.rgcn(graph_data) # (B, T, H)

# 4. Graph Transformer
graph_features = self.graph_transformer(graph_features) # (B, T, H)

# 5. rPPG信号输出
rppg = self.rppg_head(graph_features).squeeze(-1) # (B, T)

return rppg

def _build_graph(self, features: torch.Tensor) -> dict:
"""构建帧间关系图"""
B, T, D = features.shape

# 节点:每帧的特征
nodes = features # (B, T, D)

# 边关系定义
relations = {
'neighbor': [], # 邻近帧
'periodic': [], # 周期相位
'trend': [], # 趋势
'long_range': [] # 长程连接
}

for b in range(B):
for t in range(T):
# 邻近边
if t > 0:
relations['neighbor'].append((b, t, t-1))
if t < T-1:
relations['neighbor'].append((b, t, t+1))

# 周期边(假设心率60-250bpm → 周期0.24-1.5秒 → 7-45帧)
for period in [15, 20, 25, 30]: # 多尺度周期
if t + period < T:
relations['periodic'].append((b, t, t + period))

return {'nodes': nodes, 'relations': relations}


class SwinTransformer(nn.Module):
"""简化版Swin Transformer"""
def __init__(self, img_size=128, patch_size=4, in_chans=3, embed_dim=96, depths=None, num_heads=None):
super().__init__()
self.patch_embed = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)
self.norm = nn.LayerNorm(embed_dim)
self.blocks = nn.ModuleList([
nn.TransformerEncoderLayer(
d_model=embed_dim, nhead=4,
dim_feedforward=embed_dim * 4,
batch_first=True
) for _ in range(2)
])

def forward(self, x):
x = self.patch_embed(x) # (B, D, H', W')
B, D, H, W = x.shape
x = x.flatten(2).transpose(1, 2) # (B, H*W, D)
for blk in self.blocks:
x = blk(x)
x = x.transpose(1, 2).view(B, D, H, W)
return x


class RelationalGCN(nn.Module):
"""关系图卷积网络"""
def __init__(self, in_dim, hidden_dim, out_dim, num_relations, num_layers):
super().__init__()
self.num_relations = num_relations
self.layers = nn.ModuleList([
RGCNLayer(in_dim if i == 0 else hidden_dim, hidden_dim, num_relations)
for i in range(num_layers)
])

def forward(self, graph_data):
x = graph_data['nodes']
for layer in self.layers:
x = layer(x, graph_data['relations'])
return x


class RGCNLayer(nn.Module):
"""R-GCN层"""
def __init__(self, in_dim, out_dim, num_relations):
super().__init__()
self.relation_weights = nn.Parameter(
torch.randn(num_relations, in_dim, out_dim) * 0.02
)
self.self_loop = nn.Linear(in_dim, out_dim)

def forward(self, x, relations):
out = self.self_loop(x)
for rel_type, edges in relations.items():
if edges:
for b, src, dst in edges:
out[b, dst] += x[b, src] @ self.relation_weights[rel_type]
return out


class GraphTransformer(nn.Module):
"""图Transformer"""
def __init__(self, d_model, nhead, num_layers, dim_feedforward):
super().__init__()
layer = nn.TransformerEncoderLayer(
d_model=d_model, nhead=nhead,
dim_feedforward=dim_feedforward,
batch_first=True
)
self.encoder = nn.TransformerEncoder(layer, num_layers)

def forward(self, x):
return self.encoder(x)


# ============ 测试 ============
if __name__ == "__main__":
torch.manual_seed(42)

model = ReperioRPPG()

# 模拟10秒视频 @ 30fps
video = torch.randn(2, 300, 3, 128, 128) # B=2, T=300

rppg = model(video)

print("Reperio-rPPG 测试:")
print(f" 视频输入: {video.shape}")
print(f" rPPG输出: {rppg.shape}")

# 从rPPG估计心率
freqs = torch.fft.fftfreq(300, d=1/30)
psd = torch.abs(torch.fft.fft(rppg, dim=1))**2
hr_freq = freqs[torch.argmax(psd, dim=1)]
hr_bpm = hr_freq * 60

print(f" 估计心率: {hr_bpm} BPM")

params = sum(p.numel() for p in model.parameters())
print(f" 参数: {params:,} ({params/1e6:.1f}M)")

Ultra-Short rPPG (2秒估计)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
"""
Ultra-Short rPPG: 2秒视频心率估计

核心挑战:
1. 2秒仅有1-2个心跳周期 → 周期不足
2. 短信号频谱泄漏 → PSD不准

解决方案:
1. 周期性引导损失 → 强制短信号与长信号周期一致
2. 生成器重建 → 从2秒重建10秒信号
"""

class UltraShortRPPG(nn.Module):
"""
超短期rPPG估计框架

组成:
1. rPPG模型:从2秒视频提取rPPG信号
2. 生成器:从2秒rPPG重建10秒rPPG
3. 周期性引导损失:保证短/长信号周期一致
"""

def __init__(self, encoder_dim: int = 256, signal_length_2s: int = 60, signal_length_10s: int = 300):
super().__init__()

# rPPG编码器
self.rppg_encoder = nn.Sequential(
nn.Conv2d(3, 32, 3, 2, 1), nn.BatchNorm2d(32), nn.ReLU(),
nn.Conv2d(32, 64, 3, 2, 1), nn.BatchNorm2d(64), nn.ReLU(),
nn.Conv2d(64, encoder_dim, 3, 2, 1), nn.BatchNorm2d(encoder_dim), nn.ReLU(),
nn.AdaptiveAvgPool2d(1)
)

# 时序模型
self.temporal = nn.LSTM(encoder_dim, encoder_dim, batch_first=True, bidirectional=True)

# rPPG输出(2秒)
self.rppg_head_2s = nn.Linear(encoder_dim * 2, 1)

# 生成器:2秒 → 10秒
self.generator = SignalReconstructor(
input_len=signal_length_2s,
output_len=signal_length_10s,
latent_dim=encoder_dim
)

def forward(self, video_2s: torch.Tensor) -> dict:
"""
Args:
video_2s: (B, T=60, C, H, W) 2秒视频 @ 30fps
"""
B, T, C, H, W = video_2s.shape

# 空间编码
frames = video_2s.view(B * T, C, H, W)
spatial = self.rppg_encoder(frames) # (B*T, D, 1, 1)
spatial = spatial.view(B, T, -1)

# 时序建模
temporal, _ = self.temporal(spatial) # (B, T, 2D)

# 2秒rPPG信号
rppg_2s = self.rppg_head_2s(temporal).squeeze(-1) # (B, 60)

# 重建10秒rPPG
rppg_10s = self.generator(rppg_2s) # (B, 300)

return {
'rppg_2s': rppg_2s,
'rppg_10s': rppg_10s,
'temporal_features': temporal
}


class SignalReconstructor(nn.Module):
"""从2秒rPPG重建10秒信号"""

def __init__(self, input_len=60, output_len=300, latent_dim=256):
super().__init__()

# 编码2秒信号
self.encoder = nn.Sequential(
nn.Linear(input_len, latent_dim),
nn.ReLU(),
nn.Linear(latent_dim, latent_dim),
nn.ReLU()
)

# 解码10秒信号
self.decoder = nn.Sequential(
nn.Linear(latent_dim, latent_dim * 2),
nn.ReLU(),
nn.Linear(latent_dim * 2, latent_dim * 4),
nn.ReLU(),
nn.Linear(latent_dim * 4, output_len)
)

def forward(self, rppg_2s: torch.Tensor) -> torch.Tensor:
z = self.encoder(rppg_2s)
rppg_10s = self.decoder(z)
return rppg_10s


class PeriodicityGuidedLoss(nn.Module):
"""
周期性引导损失

1. 加权频谱交叉熵:短/长信号PSD一致
2. 周期相似性最大化:短信号与长信号周期一致
"""

def __init__(self, fs: int = 30, hr_range: tuple = (0.66, 4.16)):
super().__init__()
self.fs = fs
self.hr_range = hr_range

def forward(
self,
rppg_2s: torch.Tensor,
rppg_10s: torch.Tensor,
gt_10s: torch.Tensor = None
) -> Tuple[torch.Tensor, dict]:

# 频谱交叉熵
freqs = torch.fft.fftfreq(rppg_2s.size(1), d=1/self.fs)
mask = (freqs >= self.hr_range[0]) & (freqs <= self.hr_range[1])

psd_2s = torch.abs(torch.fft.fft(rppg_2s, dim=1))**2
psd_10s = torch.abs(torch.fft.fft(rppg_10s, dim=1))**2

# 归一化
psd_2s = psd_2s[:, mask]
psd_10s = psd_10s[:, mask]
psd_2s = psd_2s / (psd_2s.sum(dim=1, keepdim=True) + 1e-8)
psd_10s = psd_10s / (psd_10s.sum(dim=1, keepdim=True) + 1e-8)

spec_loss = -torch.sum(psd_2s * torch.log(psd_10s + 1e-8), dim=1).mean()

# 周期相似性
# 自相关峰值比较
autocorr_2s = self._autocorrelation(rppg_2s)
autocorr_10s = self._autocorrelation(rppg_10s)

period_loss = nn.functional.mse_loss(autocorr_2s, autocorr_10s)

# GT损失(如果有)
if gt_10s is not None:
gt_loss = nn.functional.mse_loss(rppg_10s, gt_10s)
else:
gt_loss = torch.tensor(0.0, device=rppg_2s.device)

total = spec_loss + 0.5 * period_loss + 0.3 * gt_loss

return total, {
'spec_loss': spec_loss.item(),
'period_loss': period_loss.item(),
'gt_loss': gt_loss.item()
}

def _autocorrelation(self, x):
"""简化自相关"""
n = x.size(1)
result = []
for lag in range(0, min(n, 30)):
if lag == 0:
result.append(x.var(dim=1))
else:
result.append((x[:, :-lag] * x[:, lag:]).mean(dim=1))
return torch.stack(result, dim=1)


# 测试
if __name__ == "__main__":
torch.manual_seed(42)

model = UltraShortRPPG()

# 2秒视频 @ 30fps = 60帧
video_2s = torch.randn(4, 60, 3, 128, 128)

output = model(video_2s)

print("Ultra-Short rPPG 测试:")
print(f" 2秒视频: {video_2s.shape}")
print(f" 2秒rPPG: {output['rppg_2s'].shape}")
print(f" 重建10秒rPPG: {output['rppg_10s'].shape}")

# 损失
criterion = PeriodicityGuidedLoss()
loss, metrics = criterion(output['rppg_2s'], output['rppg_10s'])

print(f"\n 损失: {loss.item():.4f}")
for k, v in metrics.items():
print(f" {k}: {v:.4f}")

# 从10秒信号估计HR
rppg_10s = output['rppg_10s'][0].detach()
freqs = torch.fft.fftfreq(300, d=1/30)
psd = torch.abs(torch.fft.fft(rppg_10s))**2
hr_idx = torch.argmax(psd)
hr_freq = freqs[hr_idx]
print(f"\n 估计心率: {hr_freq * 60:.1f} BPM")

实验结果

Reperio-rPPG 性能

数据集 场景 Reperio 最佳基线 提升
PURE 静止 0.12 MAE 0.35 -66%
PURE 旋转 0.45 MAE 0.82 -45%
PURE 说话 0.38 MAE 0.71 -46%
PURE 行走 1.82 MAE 3.15 -42%
UBFC-rPPG 综合 0.18 MAE 0.45 -60%
MMPD 自然光 2.1 MAE 4.5 -53%
MMPD 低LED 3.8 MAE 7.2 -47%
MMPD 高LED 1.9 MAE 3.8 -50%

Ultra-Short rPPG 性能

视频长度 MAE (BPM) 说明
10秒 0.82 标准基线
5秒 1.45 中等
2秒 2.31 超短
1秒 5.67 极短(不推荐)

跨域性能

训练域 测试域 2秒MAE 10秒MAE
PURE → UBFC 2.31 0.82
PURE → MMPD 4.52 2.15
UBFC → PURE 3.18 1.05
UBFC → MMPD 4.85 2.30

IMS 开发启示

1. 驾驶员心率监测方案

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# IMS rPPG 心率监测配置
ims_rppg_config = {
"sensor": "DMS IR/RGB摄像头",
"frame_rate": "30fps",
"video_length": "2秒(Ultra-Short)或10秒(标准)",
"model": "Reperio-rPPG(运动鲁棒)+ Ultra-Short(实时)",

"two_mode": {
"realtime_mode": {
"window": "2秒",
"update_rate": "每2秒更新",
"accuracy": "±2.3 BPM",
"use_case": "快速变化检测"
},
"accurate_mode": {
"window": "10秒",
"update_rate": "每5秒更新(50%重叠)",
"accuracy": "±0.8 BPM",
"use_case": "基线心率/HRV分析"
}
},

"deployment": {
"platform": "QCS8255 Hexagon NPU",
"model_size": "~15MB (INT8量化)",
"inference_time": "<50ms (2秒视频)",
"power": "<1W"
}
}

2. HRV 疲劳评估

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 从rPPG信号计算HRV → 疲劳评估
hrv_fatigue_pipeline = {
"step_1": "rPPG信号 → 心跳峰值检测",
"step_2": "RR间期序列 → HRV特征",
"step_3": "HRV特征 → 疲劳分类",

"hrv_features": {
"time_domain": ["SDNN", "RMSSD", "pNN50"],
"freq_domain": ["LF", "HF", "LF/HF ratio"],
"nonlinear": ["SampEn", "DFA α1"]
},

"fatigue_thresholds": {
"alert": "LF/HF > 1.5, RMSSD < 30ms",
"mild_fatigue": "LF/HF 0.8-1.5, RMSSD 30-50ms",
"drowsy": "LF/HF < 0.8, RMSSD > 50ms",
"microsleep_risk": "LF/HF < 0.5, RMSSD > 80ms"
},

"real_time_2s": "每2秒更新一次心率 → 检测心率骤降(疲劳信号)",
"baseline_10s": "每10秒更新HRV → 长期趋势分析"
}

3. 多模态融合架构

4. 实时性对比

方案 窗口 更新频率 精度 延迟 IMS适用
传统rPPG 10秒 5秒 ±0.8 BPM 5秒 基线建立
Ultra-Short 2秒 2秒 ±2.3 BPM 2秒 实时监测
mmWave雷达 连续 连续 ±1.8 BPM <1秒 补充验证
rPPG+雷达 2秒+连续 2秒 ±1.0 BPM 2秒 最佳融合

参考文献

  1. Reperio-rPPG: arXiv:2511.05946, GitHub
  2. Ultra-Short rPPG: arXiv:2506.22078, Pattern Recognition
  3. Mamba-rPPG: ScienceDirect, 状态空间模型
  4. PhysFormer: 先前Transformer rPPG
  5. PhysMamba: 状态空间对偶
  6. PURE/UBFC-rPPG/MMPD: 标准rPPG基准

发布时间: 2026-08-25 06:15
标签: rPPG, 心率监测, 图神经网络, Swin Transformer, R-GCN, 超短视频, 周期性, HRV, 疲劳检测, IMS