PULSE:毫米波雷达人体姿态估计的 Doppler 提示方法

ICML 2026 论文解读与代码复现

论文信息

  • 标题: Doppler-Prompting for Stable mmWave-based Human Pose Estimation
  • 作者: Shuntian Zheng, Jiaqi Li, Xiaoman Lu, Shuai He, Yu Guan
  • 机构: University of Warwick / Beijing University of Posts and Telecommunications
  • 会议: ICML 2026(唯一被接收的mmWave人体感知论文)
  • 链接: arXiv:2605.13233
  • 代码: GitHub: DandongExpress/Doppler-Prompting

核心创新

PULSE(Prompting Using Local Spectral Estimates)重新定义了 Doppler 信号在 mmWave 人体姿态估计中的角色:从对称特征通道转变为经过筛选的运动提示(motion prompt)。通过置信度门控过滤不可靠的 Doppler 响应,再通过局部交叉注意力确保只有空间一致的运动线索才能影响姿态主干网络。

一句话总结: 用一个12M参数、5.1ms推理的轻量模块,替代传统mmWave姿态估计中的Doppler特征融合方式,在三个公开数据集上全面刷新SOTA。

问题背景

mmWave 雷达姿态估计的挑战

mmWave 雷达每帧输出 Range-Angle-Doppler (RAD) 张量,包含:

  • 空间幅度(Spatial Magnitude):反射体在哪里
  • Doppler 特征(Doppler Signature):反射体如何运动
问题 描述 现有方案缺陷
Doppler 非人体专有 杂波、多径反射、硬件伪影都产生 Doppler 响应 现有方法忽略或简单融合
跨区域频谱泄漏 一个区域的 Doppler 信号泄漏到相邻区域 全局注意力导致噪声传播
时序抖动 逐帧估计导致姿态轨迹不连贯 后处理平滑(但增加延迟)

PULSE 的解决思路

方法详解

1. 双域特征构建

从 RAD 张量 $\mathbf{H}_t \in \mathbb{R}^{R \times A \times D}$ 中提取两个域:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
import torch
import torch.nn as nn
import numpy as np

class DualDomainFeatureConstructor(nn.Module):
"""
双域特征构建

输入:RAD张量 H_t (B, R, A, D)
R=距离bins, A=角度bins, D=Doppler bins

输出:
- S_t: 空间幅度 (B, R, A) - Doppler维平均
- V_t: Doppler特征 (B, R, A, D) - 原始Doppler签名
"""

def __init__(self):
super().__init__()

def forward(self, H_t: torch.Tensor) -> tuple:
"""
Args:
H_t: (B, R, A, D) RAD张量

Returns:
S_t: (B, R, A) 空间幅度图
V_t: (B, R, A, D) Doppler签名
"""
# 空间幅度:沿Doppler维取平均(或最大值)
S_t = H_t.mean(dim=-1) # (B, R, A)

# Doppler特征:保留原始Doppler维度
V_t = H_t # (B, R, A, D)

return S_t, V_t


# 测试
if __name__ == "__main__":
constructor = DualDomainFeatureConstructor()

# 模拟RAD张量 (batch=1, range=64, angle=64, doppler=16)
H_t = torch.randn(1, 64, 64, 16)

S_t, V_t = constructor(H_t)

print(f"输入 RAD: {H_t.shape}")
print(f"空间幅度 S_t: {S_t.shape}")
print(f"Doppler V_t: {V_t.shape}")

2. Tokenization

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
class RadarTokenizer(nn.Module):
"""
空间和Doppler token化

- 空间tokens: S_t分成 P_r × P_a 非重叠patch
- Doppler tokens: V_t每个range-angle cell独立embed
- 两组tokens共享相同网格
"""

def __init__(self, range_bins: int = 64, angle_bins: int = 64,
doppler_bins: int = 16, patch_size: tuple = (4, 4),
embed_dim: int = 32):
super().__init__()

self.patch_size = patch_size
self.embed_dim = embed_dim

# 空间patch embedding
self.spatial_embed = nn.Linear(
patch_size[0] * patch_size[1], embed_dim
)

# Doppler embedding(每个cell独立embed)
self.doppler_embed = nn.Linear(doppler_bins, embed_dim)

# 位置编码
n_patches_r = range_bins // patch_size[0]
n_patches_a = angle_bins // patch_size[1]
self.pos_embed = nn.Parameter(
torch.randn(1, n_patches_r * n_patches_a, embed_dim)
)

def forward(self, S_t: torch.Tensor, V_t: torch.Tensor) -> tuple:
"""
Args:
S_t: (B, R, A) 空间幅度
V_t: (B, R, A, D) Doppler签名

Returns:
spatial_tokens: (B, N, D) 空间tokens
doppler_tokens: (B, R*A, D) Doppler tokens
"""
B, R, A = S_t.shape

# 空间token化:分patch
pr, pa = self.patch_size
S_patches = S_t.unfold(1, pr, pr).unfold(2, pa, pa)
# (B, R/pr, A/pa, pr, pa)
S_patches = S_patches.contiguous().view(B, -1, pr * pa)
spatial_tokens = self.spatial_embed(S_patches) # (B, N, D)

# 添加位置编码
spatial_tokens = spatial_tokens + self.pos_embed

# Doppler token化:每个cell独立embed
V_flat = V_t.view(B, R * A, -1) # (B, R*A, D)
doppler_tokens = self.doppler_embed(V_flat) # (B, R*A, D)

return spatial_tokens, doppler_tokens


# 测试
if __name__ == "__main__":
tokenizer = RadarTokenizer(
range_bins=64, angle_bins=64, doppler_bins=16,
patch_size=(4, 4), embed_dim=32
)

S_t = torch.randn(1, 64, 64)
V_t = torch.randn(1, 64, 64, 16)

spatial_tokens, doppler_tokens = tokenizer(S_t, V_t)

print(f"空间tokens: {spatial_tokens.shape} (B, N_patches, D)")
print(f"Doppler tokens: {doppler_tokens.shape} (B, R*A, D)")
print(f"N_patches = (64/4) × (64/4) = {16*16}")

3. 置信度门控局部交叉注意力(核心模块)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
class ConfidenceGatedCrossAttention(nn.Module):
"""
置信度门控局部交叉注意力

核心设计:
1. 可学习门 g_{t,j} ∈ [0,1] 评估每个Doppler token的运动相关性
2. 空间token只关注局部邻域 N(i) 内的Doppler tokens
3. 门控权重调制注意力,抑制不可靠Doppler响应

这解决了两个关键问题:
- 跨区域频谱泄漏:局部注意力限制
- 杂波/伪影Doppler:置信度门控
"""

def __init__(self, embed_dim: int = 32, neighborhood: int = 3):
super().__init__()
self.embed_dim = embed_dim
self.neighborhood = neighborhood

# 置信度门(评估Doppler token质量)
self.gate = nn.Sequential(
nn.Linear(embed_dim, embed_dim // 2),
nn.GELU(),
nn.Linear(embed_dim // 2, 1),
nn.Sigmoid()
)

# 交叉注意力
self.q_proj = nn.Linear(embed_dim, embed_dim)
self.k_proj = nn.Linear(embed_dim, embed_dim)
self.v_proj = nn.Linear(embed_dim, embed_dim)

self.scale = embed_dim ** -0.5

def forward(self, spatial_tokens: torch.Tensor,
doppler_tokens: torch.Tensor,
n_patches_r: int, n_patches_a: int) -> torch.Tensor:
"""
Args:
spatial_tokens: (B, N, D) N = n_patches_r * n_patches_a
doppler_tokens: (B, R*A, D) 每个cell一个Doppler token
n_patches_r, n_patches_a: patch网格尺寸

Returns:
conditioned_tokens: (B, N, D) 运动条件化的空间tokens
"""
B, N, D = spatial_tokens.shape
R_A = doppler_tokens.shape[1]

# 1. 计算Doppler置信度门
gates = self.gate(doppler_tokens) # (B, R*A, 1)

# 2. 将Doppler tokens重塑为2D网格
# 假设 R*A 可以映射到 patch 网格
# 每个patch对应多个Doppler cells
cells_per_patch_r = R_A // (n_patches_r * n_patches_a) // n_patches_a + 1

# 简化:将Doppler tokens下采样到patch网格
# 实际实现中使用更精确的映射
doppler_2d = doppler_tokens.view(B, n_patches_r, -1, D)
gates_2d = gates.view(B, n_patches_r, -1, 1)

# 3. 局部交叉注意力
Q = self.q_proj(spatial_tokens) # (B, N, D)

# 为每个空间token,计算对局部Doppler tokens的注意力
# 简化实现:使用全局注意力(实际使用局部窗口)
K = self.k_proj(doppler_tokens) # (B, R*A, D)
V = self.v_proj(doppler_tokens) # (B, R*A, D)

# 注意力分数
attn = torch.bmm(Q, K.transpose(1, 2)) * self.scale # (B, N, R*A)

# 门控调制
attn = attn * gates.transpose(1, 2) # 广播门控

# 局部掩码(只允许3x3邻域)
# 实际实现使用稀疏注意力或窗口注意力
# 这里简化为全局注意力 + 门控

attn = torch.softmax(attn, dim=-1)

# 加权求和
out = torch.bmm(attn, V) # (B, N, D)

# 残差连接
conditioned = spatial_tokens + out

return conditioned


class PULSEPrompting(nn.Module):
"""
PULSE 完整模块

可作为插件替换现有backbone的前端融合
"""

def __init__(self, range_bins=64, angle_bins=64, doppler_bins=16,
patch_size=(4, 4), embed_dim=32,
neighborhood=3, beta=1.0):
super().__init__()

self.constructor = DualDomainFeatureConstructor()
self.tokenizer = RadarTokenizer(
range_bins, angle_bins, doppler_bins,
patch_size, embed_dim
)
self.cross_attn = ConfidenceGatedCrossAttention(
embed_dim, neighborhood
)

self.n_patches_r = range_bins // patch_size[0]
self.n_patches_a = angle_bins // patch_size[1]

# 输出投影
self.output_proj = nn.Linear(embed_dim, embed_dim)

def forward(self, H_t: torch.Tensor) -> torch.Tensor:
"""
Args:
H_t: (B, R, A, D) RAD张量
或 (B, T, R, A, D) 多帧序列

Returns:
spatial_tokens: (B, N, D) 运动条件化空间tokens
"""
if H_t.dim() == 5:
# 多帧模式
B, T, R, A, D = H_t.shape
# 简化:取最后一帧(实际用置信度加权聚合)
H_t = H_t[:, -1]

S_t, V_t = self.constructor(H_t)
spatial_tokens, doppler_tokens = self.tokenizer(S_t, V_t)
conditioned = self.cross_attn(
spatial_tokens, doppler_tokens,
self.n_patches_r, self.n_patches_a
)

return self.output_proj(conditioned)


# 测试
if __name__ == "__main__":
pulse = PULSEPrompting(
range_bins=64, angle_bins=64, doppler_bins=16,
patch_size=(4, 4), embed_dim=32,
neighborhood=3, beta=1.0
)

# 模拟RAD张量
H_t = torch.randn(1, 64, 64, 16)

output = pulse(H_t)

print("=== PULSE 模块 ===")
print(f"输入 RAD: {H_t.shape}")
print(f"输出 tokens: {output.shape}")

total_params = sum(p.numel() for p in pulse.parameters())
print(f"参数量: {total_params:,}")
print(f"模型大小: {total_params * 4 / 1024:.1f} KB")

4. 多帧扩展(PULSE-KF)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
class PULSEMultiFrame(nn.Module):
"""
PULSE-KF: 多帧扩展

用置信度加权的K帧Doppler聚合替代单帧Doppler tokens
不修改空间backbone
"""

def __init__(self, k_frames: int = 9, **kwargs):
super().__init__()
self.k_frames = k_frames
self.pulse = PULSEPrompting(**kwargs)

# 时序置信度聚合
self.temporal_gate = nn.Sequential(
nn.Linear(kwargs.get('embed_dim', 32), 1),
nn.Sigmoid()
)

def forward(self, H_window: torch.Tensor) -> torch.Tensor:
"""
Args:
H_window: (B, K, R, A, D) K帧RAD张量序列
Returns:
spatial_tokens: (B, N, D)
"""
B, K, R, A, D = H_window.shape

# 处理每帧
all_doppler = []
for k in range(K):
H_t = H_window[:, k]
S_t, V_t = self.pulse.constructor(H_t)
_, doppler_tokens = self.pulse.tokenizer(S_t, V_t)
all_doppler.append(doppler_tokens)

# 时序置信度加权聚合
stacked = torch.stack(all_doppler, dim=1) # (B, K, R*A, D)

weights = self.temporal_gate(stacked) # (B, K, R*A, 1)
weighted = (stacked * weights).sum(dim=1) / (weights.sum(dim=1) + 1e-8)

# 用聚合后的Doppler tokens
H_last = H_window[:, -1]
S_t, _ = self.pulse.constructor(H_last)
spatial_tokens, _ = self.pulse.tokenizer(S_t, weighted)

conditioned = self.pulse.cross_attn(
spatial_tokens, weighted,
self.pulse.n_patches_r, self.pulse.n_patches_a
)

return self.pulse.output_proj(conditioned)


# 测试
if __name__ == "__main__":
pulse_kf = PULSEMultiFrame(k_frames=9, range_bins=64, angle_bins=64,
doppler_bins=16, patch_size=(4, 4),
embed_dim=32, neighborhood=3)

# 模拟9帧序列
H_window = torch.randn(1, 9, 64, 64, 16)
output = pulse_kf(H_window)

print(f"输入 9帧RAD: {H_window.shape}")
print(f"输出 tokens: {output.shape}")

total_params = sum(p.numel() for p in pulse_kf.parameters())
print(f"参数量: {total_params:,}")

实验结果

主要结果

数据集 方法 MPJPE↓ (mm) PA-MPJPE↓ MPJVE↓ (mm/f) AKV↓
HuPR HuPRModel (SF) 65.37 58.11 14.70 14.1
MvDoppler (SF) 69.71 65.56 13.11 13.4
mmDiff (MF) 65.54 60.02 13.60 5.7
milliMamba (MF) 64.08 57.44 11.69 7.8
PULSE 1F (SF) 60.57 54.15 9.78 5.1
PULSE KF (MF) 58.64 53.01 8.16 5.0
XRF55 PULSE 1F (SF) 70.34 67.51 15.33 7.1
PULSE KF (MF) 68.99 63.81 14.05 6.5
mmRadPose PULSE 1F (SF) 68.83 60.80 12.90 6.5
PULSE KF (MF) 67.56 59.19 11.70 5.4

插件效果(替换前端融合)

Backbone 数据集 MPJPE 原始 MPJPE +PULSE MPJVE 原始 MPJVE +PULSE
mmDiff HuPR 65.54 60.89 13.60 10.07
mmDiff XRF55 79.06 72.56 21.44 16.02
milliMamba HuPR 68.08 63.43 11.69 10.47
milliMamba XRF55 74.35 68.25 17.76 14.37

关键发现: PULSE 作为插件,在不修改 backbone 的情况下,一致提升 MPJPE 和 MPJVE。

模型效率

指标 PULSE mmDiff milliMamba
参数量 12M ~24M ~18M
推理速度 5.1ms ~15ms ~8ms
FLOPs 75M ~300M ~150M
最小模型 ✅ ❌ ❌

在座舱OOP检测中的应用

mmWave 雷达 OOP 方案

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
class CabinOOPDetector:
"""
基于 PULSE + mmWave 雷达的座舱OOP检测器

优势:
1. 隐私保护(无图像)
2. 穿透毯子/衣物
3. 5.1ms推理延迟(实时)
4. 12M参数(可部署边缘NPU)
5. PULSE降低了MPJVE(抖动),适合时序稳定的OOP判断

部署方案:
- 雷达:Infineon XENSIV 60GHz 或 TI IWR6843
- 处理器:QCS8255 Hexagon NPU 或 STM32N6
"""

def __init__(self, threshold_lean: float = 30.0,
threshold_kneel: float = 0.7):
self.threshold_lean = threshold_lean # 前倾角度阈值
self.threshold_kneel = threshold_kneel # 跪立高度阈值

def detect_oop_from_joints(self, joints_3d: np.ndarray) -> dict:
"""
从3D关节坐标检测OOP

Args:
joints_3d: (J, 3) 3D关节坐标 (mm)
J=17 or 24 depending on skeleton format
Returns:
OOP检测结果
"""
# 关键关节(简化索引)
PELVIS, SPINE, NECK, HEAD = 0, 3, 12, 15
L_SHOULDER, R_SHOULDER = 16, 17
L_HIP, R_HIP = 1, 2
L_KNEE, R_KNEE = 4, 5
L_ANKLE, R_ANKLE = 7, 8

# 1. 前倾检测:躯干向量与垂直方向夹角
torso_vec = joints_3d[NECK] - joints_3d[PELVIS]
forward_angle = np.degrees(np.arctan2(
abs(torso_vec[2]), # z: 前后
max(abs(torso_vec[1]), 1.0) # y: 上下
))

# 2. 侧倾检测
side_angle = np.degrees(np.arctan2(
torso_vec[0], # x: 左右
max(abs(torso_vec[1]), 1.0)
))

# 3. 跪立检测:膝盖高度
knee_y = min(joints_3d[L_KNEE][1], joints_3d[R_KNEE][1])
hip_y = max(joints_3d[L_HIP][1], joints_3d[R_HIP][1])
is_kneeling = knee_y > hip_y * 0.5 # 膝盖接近臀部高度

results = {}
if forward_angle > self.threshold_lean:
results['forward_lean'] = {
'angle': float(forward_angle),
'severity': min(forward_angle / 60, 1.0)
}

if abs(side_angle) > self.threshold_lean:
results['side_lean'] = {
'angle': float(side_angle),
'severity': min(abs(side_angle) / 45, 1.0)
}

if is_kneeling:
results['kneeling'] = {
'knee_height_ratio': float(knee_y / (hip_y + 1e-6)),
'severity': 0.8
}

return {
'is_oop': len(results) > 0,
'oop_types': list(results.keys()),
'details': results,
'joint_metrics': {
'forward_angle': float(forward_angle),
'side_angle': float(side_angle),
'knee_hip_ratio': float(knee_y / (hip_y + 1e-6))
}
}


# 测试
if __name__ == "__main__":
detector = CabinOOPDetector()

# 模拟正常坐姿关节
normal_joints = np.array([
[0, -400, 0], # pelvis
[-100, -350, 0], # left_hip
[100, -350, 0], # right_hip
[0, -300, 0], # spine1
[-100, -200, 0], # left_knee
[100, -200, 0], # right_knee
[0, -250, 0], # spine2
[-100, -50, 0], # left_ankle
[100, -50, 0], # right_ankle
[0, -200, 0], # spine3
[-100, -50, 0], # left_foot
[100, -50, 0], # right_foot
[0, -150, 0], # neck
[-150, -180, 0], # left_collar
[150, -180, 0], # right_collar
[0, -100, 0], # head
[-200, -200, 0], # left_shoulder
[200, -200, 0], # right_shoulder
[-300, -300, 0], # left_elbow
[300, -300, 0], # right_elbow
[-400, -380, 0], # left_wrist
[400, -380, 0], # right_wrist
[-450, -380, 0], # left_hand
[450, -380, 0], # right_hand
], dtype=np.float32)

# 模拟前倾坐姿
lean_joints = normal_joints.copy()
lean_joints[NECK := 12] = [0, -100, 50] # neck前倾
lean_joints[HEAD := 15] = [0, -50, 60] # head前倾

# 模拟跪立
kneel_joints = normal_joints.copy()
kneel_joints[4] = [-100, -350, 0] # left_knee抬高
kneel_joints[5] = [100, -350, 0] # right_knee抬高

print("=== 正常坐姿 ===")
print(detector.detect_oop_from_joints(normal_joints))

print("\n=== 前倾OOP ===")
print(detector.detect_oop_from_joints(lean_joints))

print("\n=== 跪立OOP ===")
print(detector.detect_oop_from_joints(kneel_joints))

IMS 开发启示

1. mmWave OOP 技术路线

阶段 目标 技术 周期
Phase 1 PULSE 原型验证 Infineon 60GHz + PULSE开源代码 3个月
Phase 2 座舱场景适配 在实车采集RAD数据,微调PULSE 4个月
Phase 3 OOP检测集成 PULSE + CabinOOPDetector 2个月
Phase 4 边缘部署 量化INT8 + NPU部署 3个月

2. PULSE vs 其他OOP方案

方案 精度(MPJPE) 速度 隐私 成本 OOP可行性
PULSE (mmWave) ~60mm 5.1ms ✅ $30-80 ⭐⭐⭐⭐
Fast SAM 3D Body (视觉) ~50mm 65ms ❌ $300+ ⭐⭐⭐⭐
MediaPipe (视觉) ~80mm 30ms ❌ $15 ⭐⭐⭐
TAP3D (热阵列) ~待验证 ~10ms ✅ $5-15 ⭐⭐⭐

3. PULSE 作为插件的价值

PULSE 的最大价值在于插件化设计——可以替换现有mmDiff或milliMamba的前端融合模块,无需修改backbone:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 集成示例(来自论文代码)
from pulse import PULSEPrompting

# 替换现有backbone的前端
pulse_front_end = PULSEPrompting(
range_bins=64, angle_bins=64, doppler_bins=16,
patch_size=(4, 4), embed_dim=32,
neighborhood=3, beta=1.0
)

# 替换前端融合
spatial_tokens = pulse_front_end(H_t) # 输入RAD张量
# 然后送入现有backbone
output = existing_backbone(spatial_tokens)

参考资源


总结: PULSE 是 mmWave 人体姿态估计领域的重要突破——用极简的12M参数实现了SOTA精度和最低延迟。其插件化设计意味着不需要重写整个pipeline,只需替换前端融合模块即可获得提升。对 IMS 的 OOP 检测而言,PULSE + Infineon 60GHz 雷达是一个隐私友好、成本可控、精度可用的方案,值得优先评估。