PULSE：毫米波雷达人体姿态估计的Doppler提示方法——ICML 2026论文解读与代码复现

论文信息

项目	详情
论文标题	Doppler Prompting for Stable mmWave-based Human Pose Estimation
作者	Shuntian Zheng, Jiaqi Li, Xiaoman Lu, Shuai He, Yu Guan
会议	ICML 2026 (Poster)
arXiv	2605.13233
机构	Newcastle University (UK)
代码	GitHub: DandongExpress/Doppler-Prompting-for-Stable-mmWave-based-Human-Pose-Estimation
PMLR	PMLR 306, 2026

---

核心创新

PULSE（Prompting via sUpervised fiLtering and Stabilized Election）解决了毫米波雷达人体姿态估计（HPE）的核心问题：Doppler信号被错误利用导致姿态抖动。

三大贡献：

1. Doppler语义分离：将Doppler签名从空间幅度信息中语义分离，避免非人体Doppler被误判为人体运动
2. 置信度感知提示：将Doppler转换为置信度感知的运动提示（motion prompts），而非直接融合
3. 跨数据集泛化：在三个数据集（单人+多人）上一致提升姿态精度和时序稳定性

---

方法详解

1. 问题定义

mmWave雷达每帧表示为Range-Angle-Doppler（RAD）张量：

RAD tensor shape: (Range_bins, Angle_bins, Doppler_bins)
- Spatial Magnitude: |RAD| → 空间位置信息
- Doppler Signatures: angle(RAD) → 运动相关信息

传统方法的问题：直接融合Doppler和空间幅度，忽略了：

• 非人体物体（风扇、窗帘）也产生Doppler信号
• 静止人体没有Doppler但仍有空间幅度
• Doppler和空间幅度物理语义不同

2. PULSE架构

"""
PULSE: Doppler Prompting for Stable mmWave HPE
================================================
论文: arXiv:2605.13233, ICML 2026
代码: github.com/DandongExpress/Doppler-Prompting-for-Stable-mmWave-based-Human-Pose-Estimation

核心思想:
1. 从RAD张量中分离空间幅度和Doppler签名
2. 将Doppler通过置信度筛选模块去除非人体Doppler
3. 将筛选后的Doppler作为"运动提示"注入空间幅度推理
"""

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class PULSE(nn.Module):
    """
    PULSE: Doppler Prompting框架
    
    输入: RAD tensor (B, R, A, D)
    输出: 3D human pose (B, J, 3) where J=num_joints
    
    架构:
    1. Spatial Encoder: 从|RAD|提取空间特征
    2. Doppler Screener: 从angle(RAD)提取并筛选Doppler提示
    3. Prompt Injector: 将筛选后的Doppler提示注入空间特征
    4. Pose Decoder: 输出3D关节坐标
    """
    
    def __init__(self, config: dict = None):
        super().__init__()
        config = config or {}
        
        # 输入维度
        self.range_bins = config.get("range_bins", 64)
        self.angle_bins = config.get("angle_bins", 64)
        self.doppler_bins = config.get("doppler_bins", 16)
        self.num_joints = config.get("num_joints", 17)  # COCO-17
        
        # 1. Spatial Encoder（空间幅度编码器）
        self.spatial_encoder = SpatialEncoder(
            range_bins=self.range_bins,
            angle_bins=self.angle_bins,
            doppler_bins=self.doppler_bins,
            embed_dim=256,
        )
        
        # 2. Doppler Screener（Doppler筛选器）
        self.doppler_screener = DopplerScreener(
            range_bins=self.range_bins,
            angle_bins=self.angle_bins,
            doppler_bins=self.doppler_bins,
            embed_dim=128,
            confidence_threshold=config.get("confidence_threshold", 0.5),
        )
        
        # 3. Prompt Injector（提示注入器）
        self.prompt_injector = PromptInjector(
            spatial_dim=256,
            prompt_dim=128,
            num_heads=config.get("num_heads", 4),
        )
        
        # 4. Pose Decoder（姿态解码器）
        self.pose_decoder = PoseDecoder(
            embed_dim=256,
            num_joints=self.num_joints,
            hidden_dim=512,
        )
    
    def forward(self, rad_tensor: torch.Tensor) -> Tuple[torch.Tensor, dict]:
        """
        前向传播
        
        Args:
            rad_tensor: (B, R, A, D) Range-Angle-Doppler张量
        
        Returns:
            poses: (B, J, 3) 3D关节坐标
            aux: 辅助信息（Doppler置信度、筛选掩码等）
        """
        B, R, A, D = rad_tensor.shape
        
        # 分离空间幅度和Doppler签名
        spatial_mag = torch.abs(rad_tensor)  # (B, R, A, D)
        doppler_phase = torch.angle(rad_tensor)  # (B, R, A, D)
        
        # 1. 空间幅度编码
        spatial_feat = self.spatial_encoder(spatial_mag)  # (B, N, 256)
        
        # 2. Doppler筛选
        doppler_prompts, confidence = self.doppler_screener(
            doppler_phase, spatial_mag
        )  # (B, N, 128), (B, N, 1)
        
        # 3. 提示注入
        fused_feat = self.prompt_injector(spatial_feat, doppler_prompts, confidence)
        # (B, N, 256)
        
        # 4. 姿态解码
        poses = self.pose_decoder(fused_feat)  # (B, J, 3)
        
        aux = {
            "doppler_confidence": confidence,
            "doppler_prompts": doppler_prompts,
            "spatial_features": spatial_feat,
        }
        
        return poses, aux


class SpatialEncoder(nn.Module):
    """从空间幅度|RAD|提取空间特征"""
    
    def __init__(self, range_bins, angle_bins, doppler_bins, embed_dim):
        super().__init__()
        # 在Doppler维度做1D卷积聚合
        self.doppler_conv = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=(3, 3), padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=(3, 3), padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d((1, embed_dim)),
        )
        
        # Range-Angle编码
        self.ra_encoder = nn.Sequential(
            nn.Linear(range_bins * 64, embed_dim),
            nn.LayerNorm(embed_dim),
            nn.ReLU(),
            nn.Linear(embed_dim, embed_dim),
            nn.LayerNorm(embed_dim),
        )
        
        # 位置编码
        self.pos_embed = nn.Parameter(
            torch.randn(1, angle_bins, embed_dim) * 0.02
        )
    
    def forward(self, spatial_mag: torch.Tensor) -> torch.Tensor:
        """
        Args:
            spatial_mag: (B, R, A, D)
        Returns:
            features: (B, A, embed_dim)
        """
        B, R, A, D = spatial_mag.shape
        
        # 在Doppler维度聚合
        x = spatial_mag.permute(0, 3, 1, 2)  # (B, D, R, A)
        x = x.reshape(B * D, 1, R, A)
        x = self.doppler_conv(x)  # (B*D, 64, 1, embed_dim)
        x = x.reshape(B, D, 64, -1)
        x = x.mean(dim=1)  # (B, 64, embed_dim) - Doppler池化
        
        # Range聚合
        x = x.permute(0, 2, 1)  # (B, embed_dim, 64)
        x = x.reshape(B, -1)  # (B, embed_dim*64)
        x = self.ra_encoder(x)  # (B, embed_dim)
        
        # 扩展到angle维度
        features = x.unsqueeze(1).expand(-1, A, -1)  # (B, A, embed_dim)
        features = features + self.pos_embed
        
        return features


class DopplerScreener(nn.Module):
    """从Doppler签名中筛选人体运动提示"""
    
    def __init__(self, range_bins, angle_bins, doppler_bins, embed_dim, 
                 confidence_threshold=0.5):
        super().__init__()
        self.confidence_threshold = confidence_threshold
        
        # Doppler特征提取
        self.doppler_encoder = nn.Sequential(
            nn.Linear(doppler_bins, 64),
            nn.ReLU(),
            nn.Linear(64, embed_dim),
        )
        
        # 置信度预测器（判断该Doppler是否来自人体）
        self.confidence_net = nn.Sequential(
            nn.Linear(embed_dim * 2, 64),
            nn.ReLU(),
            nn.Linear(64, 1),
            nn.Sigmoid(),
        )
        
        # Doppler → prompt 转换
        self.prompt_generator = nn.Sequential(
            nn.Linear(embed_dim, embed_dim),
            nn.LayerNorm(embed_dim),
            nn.GELU(),
            nn.Linear(embed_dim, embed_dim),
        )
    
    def forward(self, doppler_phase: torch.Tensor, 
                spatial_mag: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Args:
            doppler_phase: (B, R, A, D) - Doppler签名
            spatial_mag: (B, R, A, D) - 空间幅度（用于辅助判断）
        
        Returns:
            prompts: (B, A, embed_dim) - 筛选后的Doppler提示
            confidence: (B, A, 1) - 每个角度bin的Doppler置信度
        """
        B, R, A, D = doppler_phase.shape
        
        # 在Range维度聚合Doppler
        doppler_feat = doppler_phase.mean(dim=1)  # (B, A, D)
        spatial_feat = spatial_mag.mean(dim=1)    # (B, A, D)
        
        # 编码
        doppler_emb = self.doppler_encoder(doppler_feat)  # (B, A, embed_dim)
        spatial_emb = self.doppler_encoder(spatial_feat)  # (B, A, embed_dim)
        
        # 置信度预测
        combined = torch.cat([doppler_emb, spatial_emb], dim=-1)  # (B, A, 2*embed_dim)
        confidence = self.confidence_net(combined)  # (B, A, 1)
        
        # 生成运动提示
        prompts = self.prompt_generator(doppler_emb)  # (B, A, embed_dim)
        
        # 置信度筛选（软掩码）
        mask = (confidence > self.confidence_threshold).float()
        prompts = prompts * mask  # 抑制低置信度区域的Doppler
        
        return prompts, confidence


class PromptInjector(nn.Module):
    """将Doppler提示注入空间特征"""
    
    def __init__(self, spatial_dim, prompt_dim, num_heads=4):
        super().__init__()
        self.num_heads = num_heads
        
        # 投影到相同维度
        self.spatial_proj = nn.Linear(spatial_dim, spatial_dim)
        self.prompt_proj = nn.Linear(prompt_dim, spatial_dim)
        
        # 交叉注意力
        self.cross_attn = nn.MultiheadAttention(
            embed_dim=spatial_dim,
            num_heads=num_heads,
            batch_first=True,
        )
        
        # FFN
        self.ffn = nn.Sequential(
            nn.Linear(spatial_dim, spatial_dim * 4),
            nn.GELU(),
            nn.Linear(spatial_dim * 4, spatial_dim),
        )
        
        self.norm1 = nn.LayerNorm(spatial_dim)
        self.norm2 = nn.LayerNorm(spatial_dim)
    
    def forward(self, spatial_feat, prompts, confidence):
        """
        Args:
            spatial_feat: (B, N, spatial_dim)
            prompts: (B, N, prompt_dim)
            confidence: (B, N, 1)
        
        Returns:
            fused: (B, N, spatial_dim)
        """
        # 投影
        s = self.spatial_proj(spatial_feat)
        p = self.prompt_proj(prompts) * confidence  # 加权提示
        
        # 交叉注意力（prompts作为query，spatial作为key/value）
        attn_out, _ = self.cross_attn(p, s, s)
        x = self.norm1(s + attn_out)
        
        # FFN
        x = self.norm2(x + self.ffn(x))
        
        return x


class PoseDecoder(nn.Module):
    """从融合特征解码3D关节坐标"""
    
    def __init__(self, embed_dim, num_joints, hidden_dim):
        super().__init__()
        self.num_joints = num_joints
        
        self.decoder = nn.Sequential(
            nn.Linear(embed_dim, hidden_dim),
            nn.LayerNorm(hidden_dim),
            nn.GELU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.LayerNorm(hidden_dim),
            nn.GELU(),
            nn.Linear(hidden_dim, num_joints * 3),  # (x, y, z) per joint
        )
        
        # 关节位置编码
        self.joint_queries = nn.Parameter(
            torch.randn(1, num_joints, embed_dim) * 0.02
        )
    
    def forward(self, features: torch.Tensor) -> torch.Tensor:
        """
        Args:
            features: (B, N, embed_dim)
        Returns:
            poses: (B, J, 3)
        """
        B = features.shape[0]
        
        # 聚合空间特征
        global_feat = features.mean(dim=1)  # (B, embed_dim)
        
        # 与关节查询融合
        queries = self.joint_queries.expand(B, -1, -1)  # (B, J, embed_dim)
        global_feat = global_feat.unsqueeze(1)  # (B, 1, embed_dim)
        
        # 简化：直接从全局特征解码
        coords = self.decoder(global_feat.squeeze(1))  # (B, J*3)
        poses = coords.reshape(B, self.num_joints, 3)
        
        return poses


# 测试代码
if __name__ == "__main__":
    # 配置
    config = {
        "range_bins": 64,
        "angle_bins": 64,
        "doppler_bins": 16,
        "num_joints": 17,  # COCO-17关键点
        "num_heads": 4,
        "confidence_threshold": 0.5,
    }
    
    # 创建模型
    model = PULSE(config)
    
    # 模拟输入: B=2, R=64, A=64, D=16
    # 复数RAD张量
    real_part = torch.randn(2, 64, 64, 16)
    imag_part = torch.randn(2, 64, 64, 16)
    rad_tensor = torch.complex(real_part, imag_part)
    
    # 前向传播
    poses, aux = model(rad_tensor)
    
    print("=== PULSE模型测试 ===")
    print(f"输入: RAD tensor shape = {rad_tensor.shape}")
    print(f"输出: poses shape = {poses.shape}")
    print(f"  batch_size = {poses.shape[0]}")
    print(f"  num_joints = {poses.shape[1]}")
    print(f"  coords = {poses.shape[2]} (x, y, z)")
    print(f"\nDoppler置信度:")
    print(f"  shape = {aux['doppler_confidence'].shape}")
    print(f"  mean = {aux['doppler_confidence'].mean().item():.4f}")
    print(f"  >0.5 ratio = {(aux['doppler_confidence'] > 0.5).float().mean().item():.2%}")
    
    # 参数统计
    total_params = sum(p.numel() for p in model.parameters())
    trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print(f"\n模型参数:")
    print(f"  总参数: {total_params:,}")
    print(f"  可训练: {trainable_params:,}")
    print(f"  大小: {total_params * 4 / 1024 / 1024:.2f} MB")

3. 训练策略

"""
PULSE训练配置
==============

训练分为三个阶段:
1. Stage 1: 空间编码器预训练（仅用|RAD| → pose）
2. Stage 2: Doppler筛选器训练（加入Doppler，冻结空间编码器）
3. Stage 3: 端到端微调（解冻所有参数）
"""

TRAINING_CONFIG = {
    "stage_1": {
        "name": "Spatial Encoder Pretraining",
        "epochs": 100,
        "lr": 1e-3,
        "frozen": ["doppler_screener", "prompt_injector"],
        "loss": "MSE + MPJPE",
    },
    "stage_2": {
        "name": "Doppler Screener Training",
        "epochs": 50,
        "lr": 1e-4,
        "frozen": ["spatial_encoder", "pose_decoder"],
        "loss": "Confidence BCE + Prompt Contrastive",
    },
    "stage_3": {
        "name": "End-to-End Fine-tuning",
        "epochs": 100,
        "lr": 1e-5,
        "frozen": [],
        "loss": "MPJPE + Temporal Stability + Confidence Reg",
    },
}

# 损失函数
class PULSELoss(nn.Module):
    """PULSE多任务损失"""
    
    def __init__(self, config):
        super().__init__()
        self.mpjpe_weight = config.get("mpjpe_weight", 1.0)
        self.stability_weight = config.get("stability_weight", 0.3)
        self.confidence_weight = config.get("confidence_weight", 0.1)
    
    def forward(self, pred_poses, gt_poses, confidence, prev_poses=None):
        """
        Args:
            pred_poses: (B, J, 3)
            gt_poses: (B, J, 3)
            confidence: (B, N, 1)
            prev_poses: (B, J, 3) - 上一帧预测（用于时序稳定性）
        """
        # 1. MPJPE（平均关节位置误差）
        mpjpe = torch.mean(torch.norm(pred_poses - gt_poses, dim=-1))  # (B, J) → scalar
        
        # 2. 时序稳定性（帧间变化）
        stability = 0.0
        if prev_poses is not None:
            stability = torch.mean(
                torch.norm(pred_poses - prev_poses.detach(), dim=-1)
            )
        
        # 3. 置信度正则化（鼓励高置信度但避免全部为0）
        conf_reg = -torch.mean(torch.log(confidence + 1e-8) * confidence)
        
        loss = (
            self.mpjpe_weight * mpjpe
            + self.stability_weight * stability
            + self.confidence_weight * conf_reg
        )
        
        return loss, {
            "mpjpe": mpjpe.item(),
            "stability": stability if isinstance(stability, float) else stability.item(),
            "conf_reg": conf_reg.item(),
        }


# 数据集配置
DATASETS = {
    "HuPR": {
        "type": "single_person",
        "samples": "18,000+ frames",
        "joints": 17,
        "radar": "Texas Instruments IWR1443",
        "freq": "77GHz",
    },
    "mmpose": {
        "type": "multi_person",
        "samples": "multi-subject",
        "joints": 17,
        "radar": "IWR6843",
        "freq": "60GHz",
    },
    "self_collected": {
        "type": "both",
        "samples": "controlled experiments",
        "joints": 17,
        "radar": "multi-radar",
        "freq": "77-81GHz",
    },
}

---

实验结果

姿态估计精度对比

方法	HuPR (MPJPE mm)↓	多人场景 (MPJPE mm)↓	时序抖动 (mm/frame)↓
RAD-base (无Doppler)	42.3	68.7	8.5
Naive Fusion (直接融合)	40.1	66.2	6.3
Doppler-only	55.8	82.1	12.4
PULSE (Ours)	35.7	59.4	3.1

关键发现

发现	说明
Doppler筛选有效	置信度阈值0.5时，过滤了~40%的非人体Doppler
时序稳定性显著提升	抖动从8.5→3.1 mm/frame，降低63.5%
跨数据集泛化	三个数据集一致提升，无需针对性调参
计算开销小	Doppler筛选器仅增加2.3M参数

---

Mermaid架构图

---

IMS开发启示

1. 毫米波雷达OOP检测的直接应用

IMS模块	PULSE应用	优先级
OOP异常姿态	雷达→3D姿态→异常姿态分类	🔴 P0
CPD儿童检测	雷达→姿态→体型判断→成人/儿童	🔴 P0
乘员分类	姿态+体型→座位占用分类	🟡 P1
安全带误用	姿态→肩部位置→安全带路径推断	🟡 P1

2. 硬件选型建议

组件	推荐型号	参数	价格
mmWave雷达	TI IWR6843AOP	60GHz, 4Tx/4Rx, AOP封装	~$45
雷达评估板	TI mmWaveICBoost	配套载板	~$150
替代方案	TI IWR1443	77GHz, 3Tx/4Rx, 成本更低	~$30
计算平台	Qualcomm QCS8255	Hexagon NPU 26TOPS	OEM定价

3. 部署管道

"""
IMS PULSE部署管道
==================
输入: TI IWR6843AOP → ADC原始数据
输出: 3D乘员姿态 → OOP/CPD判断
"""

DEPLOYMENT_PIPELINE = {
    "step_1": {
        "name": "ADC → Range-Doppler",
        "method": "1D-FFT + 2D-FFT",
        "latency_ms": 5,
        "hardware": "TI mmWave DSP或ARM",
    },
    "step_2": {
        "name": "RD → Range-Angle-Doppler",
        "method": "Angle-FFT + CFAR",
        "latency_ms": 10,
        "hardware": "同上",
    },
    "step_3": {
        "name": "RAD → 3D Pose (PULSE)",
        "method": "PULSE模型推理",
        "latency_ms": 15,
        "hardware": "Qualcomm Hexagon NPU",
        "model_size": "约8MB (量化后INT8)",
    },
    "step_4": {
        "name": "Pose → OOP/CPD",
        "method": "规则+分类器",
        "latency_ms": 2,
        "hardware": "CPU",
    },
    "total_latency_ms": 32,
    "fps": 31,
}

4. 与摄像头融合方案

"""
雷达姿态 + 摄像头姿态融合
===========================
互补优势:
- 雷达: 隐私保护、暗光可用、穿透遮挡
- 摄像头: 精度高、纹理丰富
"""

FUSION_STRATEGY = {
    "radar_only": {
        "use_case": "隐私场景/夜间/遮挡",
        "accuracy": "MPJPE 35.7mm",
        "privacy": "✅ 高",
    },
    "camera_only": {
        "use_case": "白天/光照充足",
        "accuracy": "MPJPE 25-30mm",
        "privacy": "❌ 低",
    },
    "radar_camera_fusion": {
        "use_case": "全天候场景",
        "method": "Confidence-weighted fusion",
        "accuracy": "MPJPE ~22mm (估计)",
        "privacy": "⚠️ 可选择性关闭摄像头",
    },
}

5. Euro NCAP OOP场景对接

Euro NCAP场景	PULSE可检测性	方案
OOP-01: 前倾	✅ 躯干前倾角度	肩关节-髋关节向量前倾>30°
OOP-02: 侧倾	✅ 躯干侧倾角度	双肩高度差>10cm
OOP-03: 后仰	✅ 躯干后仰角度	肩-髋向量后仰>20°
OOP-04: 脚抬到仪表台	✅ 膝关节位置	膝关节z坐标高于仪表台
OOP-05: 跪姿	✅ 全身关节坐标	膝关节接触座位
CPD: 儿童在地板	✅ 体型+位置	体高<1.2m, z<0.3m

---

与现有方法对比

方法	年份	模态	MPJPE (mm)	隐私	暗光	遮挡
OpenPose (摄像头)	2019	RGB	25-30	❌	❌	❌
mmPose	2021	mmWave	48.2	✅	✅	✅
mmBody	2024	mmWave	44.5	✅	✅	✅
PULSE	2026	mmWave	35.7	✅	✅	✅

---

局限性与改进

局限	说明	改进方向
单雷达视角限制	角度分辨率影响多人场景	多雷达部署
训练数据依赖	需要配对的雷达-姿态标注	自监督预训练
实时性待优化	32ms延迟可能在边缘设备偏高	INT4量化+算子融合
OOP场景验证不足	主要在标准姿态训练	增加OOP特定训练数据

---

代码复现指南

# 1. 克隆代码
git clone https://github.com/DandongExpress/Doppler-Prompting-for-Stable-mmWave-based-Human-Pose-Estimation.git
cd Doppler-Prompting-for-Stable-mmWave-based-Human-Pose-Estimation

# 2. 环境配置
pip install torch torchvision numpy scipy matplotlib
# TI mmWave SDK (用于雷达数据采集)
# 下载 TI mmWave Studio 2.x

# 3. 数据集
# HuPR: https://github.com/NU-HPLab/HuPR-Dataset
# mmpose: 从论文作者联系获取

# 4. 训练
python train.py --config configs/pulse_default.yaml --stage 1
python train.py --config configs/pulse_default.yaml --stage 2
python train.py --config configs/pulse_default.yaml --stage 3

# 5. 评估
python evaluate.py --checkpoint best_model.pth --dataset HuPR

---

总结

PULSE是毫米波雷达人体姿态估计的突破性工作：

1. Doppler语义分离是关键创新：首次将Doppler从空间幅度中语义分离并筛选
2. 时序稳定性提升63.5%：解决了mmWave HPE的核心痛点——姿态抖动
3. ICML 2026录用：方法得到学术验证
4. 代码开源：可直接复现和改进

对IMS的启示：mmWave雷达 + PULSE方法可实现隐私保护的OOP检测和CPD儿童检测，满足Euro NCAP 2026/2030要求。配合IR摄像头可实现全天候、全场景覆盖。