Reperio-rPPG：图神经网络建模心跳周期性，座舱非接触心率检测新SOTA

论文信息

项目 内容
标题 Reperio-rPPG: Relational Temporal Graph Neural Networks for Periodicity Learning in Remote Physiological Measurement
作者 Ba-Thinh Nguyen, Thach-Ha Ngoc Pham, Hoang-Long Duc Nguyen, Thi-Duyen Ngo, Thanh-Ha Le
机构 越南国立大学工程技术大学 (VNU UET)
arXiv 2511.05946
代码 github.com/deconasser/Reperio-rPPG
数据集 PURE, UBFC-rPPG, MMPD

核心创新

Reperio-rPPG 首次将 关系图卷积网络 (R-GCN) 与 Graph Transformer 结合，显式建模 rPPG 信号的准周期性结构。传统方法将血容量脉冲(BVP)视为普通时序信号，忽略了心跳的周期性本质——每个周期内有 intra-cycle 结构，周期之间有 inter-cycle 一致性。

关键洞察

维度 传统方法 Reperio-rPPG
时序建模 CNN/Transformer 通用时序 图结构显式建模周期性
帧间关系 固定位移或注意力 自适应关系图
周期捕获 隐式学习 显式建模 intra/inter-cycle
增强策略 常规增强 CutMix + NDF + MPOS

方法详解

1. 整体架构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
"""
Reperio-rPPG: 关系时序图神经网络用于远程生理测量
论文: arXiv:2511.05946
核心: Swin Transformer + R-GCN + Graph Transformer
"""

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class ReperioRPPG(nn.Module):
    """
    Reperio-rPPG 完整模型
    
    Pipeline:
    1. Swin Transformer 提取空间特征
    2. Temporal Patch Shift (TPS) 时序交互
    3. R-GCN 建模帧间周期关系
    4. Graph Transformer 捕获长程周期依赖
    5. 多尺度 POS + NDF 信号提取
    """
    
    def __init__(self, config: dict):
        super().__init__()
        # Stage 1: 空间特征提取 (Swin Transformer)
        self.spatial_encoder = SwinTransformer(
            img_size=config['img_size'],
            patch_size=config['patch_size'],
            in_chans=3,
            embed_dim=config['embed_dim'],
            depths=[2, 2, 6, 2],
            num_heads=[3, 6, 12, 24],
            window_size=7
        )
        
        # Stage 2: 时序位移 (TPS)
        self.tps = TemporalPatchShift(
            shift_ratio=config.get('tps_ratio', 0.1)
        )
        
        # Stage 3: 关系图卷积网络
        self.rgcn = RelationalGCN(
            in_features=config['embed_dim'],
            hidden_features=config['hidden_dim'],
            num_layers=config.get('rgcn_layers', 3),
            num_relations=config.get('num_relations', 4)
        )
        
        # Stage 4: Graph Transformer
        self.graph_transformer = GraphTransformer(
            in_features=config['hidden_dim'],
            d_model=config['hidden_dim'],
            nhead=config.get('nhead', 8),
            num_layers=config.get('gt_layers', 4),
            dim_feedforward=config['hidden_dim'] * 4
        )
        
        # Stage 5: 信号输出头
        self.signal_head = nn.Sequential(
            nn.Linear(config['hidden_dim'], config['hidden_dim'] // 2),
            nn.GELU(),
            nn.Linear(config['hidden_dim'] // 2, 1)
        )
        
    def forward(self, video: torch.Tensor) -> torch.Tensor:
        """
        前向传播
        
        Args:
            video: 输入视频 (B, T, C, H, W)
                   B=batch, T=时间帧数
        
        Returns:
            bvp_signal: BVP 信号 (B, T)
        """
        B, T, C, H, W = video.shape
        
        # Stage 1: 逐帧空间特征
        frames = video.view(B * T, C, H, W)
        spatial_features = self.spatial_encoder(frames)  # (B*T, D)
        spatial_features = spatial_features.view(B, T, -1)
        
        # Stage 2: 时序位移
        shifted = self.tps(spatial_features)
        
        # Stage 3: 构建关系图并 R-GCN
        # 帧作为节点，关系边编码时序+周期关系
        graph = self._build_temporal_graph(T, device=video.device)
        rgcn_out = self.rgcn(shifted, graph)  # (B, T, D')
        
        # Stage 4: Graph Transformer 长程依赖
        gt_out = self.graph_transformer(rgcn_out, graph)
        
        # Stage 5: BVP 信号预测
        bvp = self.signal_head(gt_out).squeeze(-1)  # (B, T)
        
        return bvp
    
    def _build_temporal_graph(self, T: int, device: torch.device):
        """
        构建时序关系图
        
        边类型:
        - relation 0: 相邻帧 (短期)
        - relation 1: 同周期帧 (intra-cycle)
        - relation 2: 跨周期帧 (inter-cycle)
        - relation 3: 长程依赖
        """
        edges = []
        for i in range(T):
            # 相邻帧
            if i > 0:
                edges.append([i, i-1, 0])
            if i < T - 1:
                edges.append([i, i+1, 0])
            
            # 同周期帧 (假设 ~15帧/周期 @30fps, HR=120bpm)
            for offset in [2, 4, 7, 10, 15, 20, 30]:
                j = i + offset
                if 0 <= j < T:
                    rel = 1 if offset <= 15 else 2
                    edges.append([i, j, rel])
            
            # 长程依赖
            for offset in [45, 60, 90]:
                j = i + offset
                if 0 <= j < T:
                    edges.append([i, j, 3])
        
        edge_index = torch.tensor([[e[0], e[1]] for e in edges], 
                                   device=device).t().contiguous()
        edge_type = torch.tensor([e[2] for e in edges], 
                                  device=device)
        return {'edge_index': edge_index, 'edge_type': edge_type}


class TemporalPatchShift(nn.Module):
    """时序补丁位移 - 轻量级时序交互"""
    
    def __init__(self, shift_ratio: float = 0.1):
        super().__init__()
        self.shift_ratio = shift_ratio
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        Args:
            x: (B, T, D)
        Returns:
            shifted: (B, T, D)
        """
        shifted = x.clone()
        shift_dim = int(x.size(-1) * self.shift_ratio)
        if shift_dim > 0:
            # 前向位移
            shifted[:, 1:, :shift_dim] = x[:, :-1, :shift_dim]
            # 后向位移
            shifted[:, :-1, shift_dim:2*shift_dim] = x[:, 1:, shift_dim:2*shift_dim]
        return shifted


class SwinTransformer(nn.Module):
    """Swin Transformer 空间特征提取 (简化版)"""
    
    def __init__(self, **kwargs):
        super().__init__()
        self.proj = nn.Conv2d(3, kwargs.get('embed_dim', 96), 
                               kernel_size=4, stride=4)
        self.embed_dim = kwargs.get('embed_dim', 96)
        
    def forward(self, x):
        x = self.proj(x)  # (B, D, H/4, W/4)
        x = x.flatten(2).transpose(1, 2)  # (B, N, D)
        return x.mean(dim=1)  # (B, D) 全局池化


class RelationalGCN(nn.Module):
    """关系图卷积网络"""
    
    def __init__(self, in_features, hidden_features, num_layers, num_relations):
        super().__init__()
        self.layers = nn.ModuleList([
            RGCNLayer(in_features if i == 0 else hidden_features,
                       hidden_features, num_relations)
            for i in range(num_layers)
        ])
    
    def forward(self, x, graph):
        for layer in self.layers:
            x = layer(x, graph)
        return x


class RGCNLayer(nn.Module):
    def __init__(self, in_f, out_f, num_rel):
        super().__init__()
        self.weight = nn.Parameter(torch.Tensor(num_rel, in_f, out_f))
        nn.init.xavier_uniform_(self.weight)
    
    def forward(self, x, graph):
        # 简化实现
        out = torch.zeros_like(x)
        for r in range(self.weight.size(0)):
            mask = (graph['edge_type'] == r)
            if mask.any():
                idx = graph['edge_index'][:, mask]
                msg = x[idx[0]] @ self.weight[r]
                out.scatter_add_(0, idx[1].unsqueeze(1).expand(-1, out.size(1)), msg)
        return F.normalize(out, dim=-1)


class GraphTransformer(nn.Module):
    """Graph Transformer for long-range periodic dependencies"""
    
    def __init__(self, in_features, d_model, nhead, num_layers, dim_feedforward):
        super().__init__()
        self.pos_embed = nn.Parameter(torch.randn(1, 300, d_model) * 0.02)
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model, nhead=nhead,
            dim_feedforward=dim_feedforward,
            batch_first=True, dropout=0.1
        )
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers)
    
    def forward(self, x, graph=None):
        T = x.size(1)
        x = x + self.pos_embed[:, :T]
        return self.transformer(x)


# 测试
if __name__ == "__main__":
    config = {
        'img_size': 224,
        'patch_size': 4,
        'embed_dim': 96,
        'hidden_dim': 256,
        'rgcn_layers': 3,
        'num_relations': 4,
        'nhead': 8,
        'gt_layers': 4,
        'tps_ratio': 0.1
    }
    
    model = ReperioRPPG(config)
    
    # 模拟视频输入: 2 batch, 150 帧, 3 通道, 224x224
    video = torch.randn(2, 150, 3, 224, 224)
    bvp = model(video)
    
    print(f"Input shape: {video.shape}")
    print(f"BVP output shape: {bvp.shape}")
    print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

2. 增强策略

Reperio-rPPG 组合了三种增强技术：

策略 全称 作用
TCM Temporal CutMix 时间维度混合帧，提升泛化
NDF Normalized Difference Frames 差帧归一化，增强脉冲信号
MPOS Multi-Scale Plane-Orthogonal-to-Skin 多尺度正交投影，分离皮肤信号 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
def temporal_cutmix(x: torch.Tensor, alpha: float = 0.2):
    """
    时序 CutMix 增强
    
    Args:
        x: (B, T, D) 时序特征
        alpha: 混合比例
    """
    B, T, D = x.shape
    lam = torch.distributions.Beta(alpha, alpha).sample((B, 1, 1))
    lam = lam.to(x.device)
    
    # 随机选择混合时间点
    t = torch.randint(1, T, (B,)).to(x.device)
    
    # 打乱索引
    perm = torch.randperm(B)
    
    x_mix = x.clone()
    for i in range(B):
        x_mix[i, t[i]:] = lam[i] * x[i, t[i]:] + (1 - lam[i]) * x[perm[i], t[i]:]
    
    return x_mix

3. 基准测试结果

数据集 条件 方法 HR MAE ↓ RMSE ↓
PURE 静止 PhysFormer 0.85 1.23
RADIANT 0.72 1.01
Reperio 0.41 0.58
PURE 旋转 PhysFormer 3.21 4.52
Reperio 1.83 2.41
PURE 行走 PhysFormer 8.75 12.3
Reperio 4.92 6.87
UBFC-rPPG 室内 PhysFormer 0.92 1.35
Reperio 0.53 0.79
MMPD 自然光 PhysFormer 5.43 7.82
Reperio 3.12 4.51
MMPD 低LED PhysFormer 9.87 14.2
Reperio 5.64 8.03

MAE = Mean Absolute Error (BPM), RMSE = Root Mean Square Error (BPM)

IMS 开发启示

1. 座舱 rPPG 方案对比

方案 精度 (静止) 精度 (运动) 延迟 适用场景
接触式 PPG ±1 BPM ±2 BPM 实时 方向盘传感器
传统 rPPG ±3 BPM ±8 BPM 1-3s 固定摄像头
PhysFormer ±1.2 BPM ±8.7 BPM 0.5s 高算力平台
Reperio-rPPG ±0.6 BPM ±5 BPM 0.3s 可部署边缘

2. 座舱部署架构

graph LR
    A[座舱摄像头<br/>RGB 30fps] --> B[人脸检测+ROI]
    B --> C[Swin Transformer<br/>空间特征提取]
    C --> D[TPS 时序位移]
    D --> E[R-GCN 周期图]
    E --> F[Graph Transformer<br/>长程依赖]
    F --> G[BVP 信号]
    G --> H[FFT 心率估计]
    H --> I[HR/HRV/呼吸率]
    
    style A fill:#e1f5fe
    style I fill:#c8e6c9

3. 部署优化建议

优化项 方法 预期效果
模型量化 INT8 量化 Swin 4x 加速, <2% 精度损失
图稀疏化 减少边类型数量 2x 加速 R-GCN
窗口调整 3s → 5s 窗口 精度提升,延迟增加
多模态融合 rPPG + mmWave 互补,精度提升 30%

4. Euro NCAP 关联

Euro NCAP 2026 OMS 要求中，乘员生理监测是加分项。Reperio-rPPG 可直接用于：

功能 场景 NCAP 关联
驾驶员心率 疲劳/压力评估 DSM 加分项
乘员心率 健康监测 OMS 加分项
呼吸率 呼吸异常检测 CPD 辅助
HRV 压力/自主神经平衡 高级健康监测

技术路线定位

graph TD
    A[传统信号处理<br/>BSS/CHROM/POS] --> B[CNN<br/>DeepPhys/TS-CAN]
    B --> C[3D CNN<br/>PhysNet]
    C --> D[Transformer<br/>PhysFormer/RADIANT]
    D --> E[图神经网络<br/>Reperio-rPPG]
    E --> F[未来: 多模态融合<br/>rPPG+雷达+EEG]
    
    style E fill:#fff3e0,stroke:#ff6f00

总结

Reperio-rPPG 是 rPPG 领域首次成功将图神经网络应用于心跳周期性建模的工作。对 IMS 开发的关键价值：

  1. 运动鲁棒性大幅提升 — 行走场景 MAE 从 8.75 → 4.92 BPM，使车载场景可行
  2. 图结构天然适配周期性 — 帧作为节点，周期关系作为边，物理意义明确
  3. 开源代码可用 — 可直接复现和集成
  4. 计算量可控 — Swin + R-GCN + Graph Transformer 可量化部署

论文推荐指数：⭐⭐⭐⭐⭐ (5/5)

对座舱非接触心率监测有直接应用价值，建议在 IMS 下一版本中集成测试。

https://dapalm.com/2026/08/22/2026-08-22-reperio-rppg-graph-neural-network-periodicity-cabin-heart-rate/
作者
Mars
发布于
2026年8月22日
许可协议