车辆乘员 3D 姿态估计论文解读与代码复现

论文信息

• 论文标题: Three-Dimensional Posture Estimation of Vehicle Occupants Using Depth and Infrared Images
• 期刊: Sensors 2024, 24(17), 5530
• DOI: 10.3390/s24175530
• 发表时间: 2024年8月27日
• 开源地址: https://doi.org/10.3390/s24175530

---

核心创新

本研究提出了一种 融合深度图和红外图像的车辆乘员 3D 姿态估计方法，解决了传统 RGB 方法在光照变化、遮挡、隐私保护方面的局限性。

关键突破：

1. 跨模态自监督学习：利用合成数据预训练，真实数据微调，减少标注成本
2. 自适应融合网络：动态融合深度和红外特征，提升遮挡场景鲁棒性
3. 轻量化部署：模型压缩至 15MB，边缘设备实时推理 > 25fps

---

问题定义

Euro NCAP 2026 OOP 检测要求

Euro NCAP 2026 要求检测 Out-of-Position（OOP）异常姿态，包括：

OOP 场景	描述	检测难度
向前倾斜	身体前倾超过安全距离	⭐⭐
侧向倾斜	身体侧倾超出座椅边缘	⭐⭐⭐
头部异常位置	头部靠近仪表盘或侧窗	⭐⭐⭐⭐
腿部位姿异常	腿部伸出或翘起	⭐⭐⭐
躺卧姿态	座椅完全放平	⭐⭐⭐⭐⭐

传统 RGB 方法局限性

问题	RGB 方法表现	影响
光照变化	性能下降 30-40%	夜间、隧道场景失效
遮挡	无法推理隐藏关节	穿厚重衣物时失效
隐私保护	录制面部图像	数据合规风险
深度信息缺失	仅 2D 坐标	无法判断 OOP 临界距离

---

方法详解

1. 系统架构

graph TB
    subgraph 输入层
        A1[深度摄像头<br/>Time-of-Flight]
        A2[红外摄像头<br/>IR 940nm]
    end
    
    subgraph 特征提取层
        B1[深度编码器<br/>ResNet-18]
        B2[红外编码器<br/>MobileNetV3]
    end
    
    subgraph 融合层
        C1[跨模态注意力<br/>Cross-Modal Attention]
        C2[自适应权重融合<br/>Adaptive Weighted Fusion]
    end
    
    subgraph 姿态估计层
        D1[3D 关键点回归<br/>17 joints × 3D]
        D2[姿态分类头<br/>Normal/OOP/Unknown]
    end
    
    subgraph 输出层
        E1[关节位置<br/>精度 < 10cm]
        E2[姿态分类<br/>准确率 > 92%]
    end
    
    A1 --> B1
    A2 --> B2
    B1 --> C1
    B2 --> C1
    C1 --> C2
    C2 --> D1
    C2 --> D2
    D1 --> E1
    D2 --> E2

2. 跨模态注意力融合

import torch
import torch.nn as nn
import torch.nn.functional as F

class CrossModalAttentionFusion(nn.Module):
    """跨模态注意力融合模块
    
    参考：Tambwekar et al., Sensors 2024
    
    核心思想：
    - 深度特征提供几何信息（3D 结构）
    - 红外特征提供纹理信息（边缘、轮廓）
    - 注意力机制动态调整融合权重
    """
    
    def __init__(self, depth_channels=256, ir_channels=128, hidden_dim=64):
        super().__init__()
        
        # 深度特征投影
        self.depth_proj = nn.Sequential(
            nn.Conv2d(depth_channels, hidden_dim, 1),
            nn.BatchNorm2d(hidden_dim),
            nn.ReLU(inplace=True)
        )
        
        # 红外特征投影
        self.ir_proj = nn.Sequential(
            nn.Conv2d(ir_channels, hidden_dim, 1),
            nn.BatchNorm2d(hidden_dim),
            nn.ReLU(inplace=True)
        )
        
        # 注意力权重生成
        self.attention_conv = nn.Sequential(
            nn.Conv2d(hidden_dim * 2, hidden_dim, 3, padding=1),
            nn.BatchNorm2d(hidden_dim),
            nn.ReLU(inplace=True),
            nn.Conv2d(hidden_dim, 2, 1),  # 输出 2 个权重（深度、红外）
            nn.Softmax(dim=1)
        )
        
    def forward(self, depth_feat, ir_feat):
        """
        Args:
            depth_feat: 深度特征, shape=(B, C_d, H, W)
            ir_feat: 红外特征, shape=(B, C_ir, H, W)
            
        Returns:
            fused_feat: 融合特征, shape=(B, hidden_dim, H, W)
        """
        # 特征对齐（空间尺寸）
        if depth_feat.shape[2:] != ir_feat.shape[2:]:
            ir_feat = F.interpolate(ir_feat, size=depth_feat.shape[2:], 
                                    mode='bilinear', align_corners=False)
        
        # 投影到统一维度
        depth_proj = self.depth_proj(depth_feat)  # (B, 64, H, W)
        ir_proj = self.ir_proj(ir_feat)           # (B, 64, H, W)
        
        # 拼接
        concat_feat = torch.cat([depth_proj, ir_proj], dim=1)  # (B, 128, H, W)
        
        # 计算注意力权重
        attention_weights = self.attention_conv(concat_feat)  # (B, 2, H, W)
        w_depth = attention_weights[:, 0:1, :, :]  # (B, 1, H, W)
        w_ir = attention_weights[:, 1:2, :, :]     # (B, 1, H, W)
        
        # 加权融合
        fused_feat = w_depth * depth_proj + w_ir * ir_proj
        
        return fused_feat, attention_weights


# 实际测试
if __name__ == "__main__":
    # 模拟输入
    batch_size = 2
    depth_feat = torch.randn(batch_size, 256, 32, 32)
    ir_feat = torch.randn(batch_size, 128, 32, 32)
    
    # 融合
    fusion_module = CrossModalAttentionFusion()
    fused_feat, weights = fusion_module(depth_feat, ir_feat)
    
    print(f"融合特征维度: {fused_feat.shape}")
    print(f"注意力权重维度: {weights.shape}")
    print(f"深度权重范围: [{weights[:, 0].min():.3f}, {weights[:, 0].max():.3f}]")
    print(f"红外权重范围: [{weights[:, 1].min():.3f}, {weights[:, 1].max():.3f}]")

3. 3D 关键点回归头

class Occupant3DPoseHead(nn.Module):
    """乘员 3D 姿态估计头
    
    输出：
    - 17 个身体关节的 3D 坐标（x, y, z）
    - 每个关节的可见性置信度
    - 整体姿态分类（Normal/OOP）
    """
    
    # 身体关节定义（参考 COCO 17 关键点）
    KEYPOINT_NAMES = [
        'nose', 'left_eye', 'right_eye', 'left_ear', 'right_ear',  # 0-4
        'left_shoulder', 'right_shoulder',                         # 5-6
        'left_elbow', 'right_elbow',                               # 7-8
        'left_wrist', 'right_wrist',                               # 9-10
        'left_hip', 'right_hip',                                   # 11-12
        'left_knee', 'right_knee',                                 # 13-14
        'left_ankle', 'right_ankle'                                # 15-16
    ]
    
    # OOP 检测关键关节
    OOP_CRITICAL_JOINTS = [0, 5, 6, 11, 12]  # 鼻子、肩膀、髋部
    
    def __init__(self, in_channels=64, num_joints=17):
        super().__init__()
        
        # 3D 坐标回归
        self.joint_regressor = nn.Sequential(
            nn.Conv2d(in_channels, 256, 3, padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, num_joints * 3, 1)  # 17 × 3 = 51
        )
        
        # 可见性置信度
        self.visibility_head = nn.Sequential(
            nn.Conv2d(in_channels, 64, 3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(64, num_joints, 1),
            nn.Sigmoid()
        )
        
        # 姿态分类
        self.pose_classifier = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),
            nn.Flatten(),
            nn.Linear(in_channels, 128),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(128, 3)  # Normal, OOP, Unknown
        )
        
    def forward(self, x):
        """
        Args:
            x: 融合特征, shape=(B, C, H, W)
            
        Returns:
            joints_3d: 3D 关节坐标, shape=(B, 17, 3)
            visibility: 可见性, shape=(B, 17)
            pose_class: 姿态分类, shape=(B, 3)
        """
        B = x.shape[0]
        
        # 3D 坐标回归
        joint_heatmaps = self.joint_regressor(x)  # (B, 51, H, W)
        
        # 全局平均池化获取坐标
        joint_coords = F.adaptive_avg_pool2d(joint_heatmaps, 1).squeeze(-1).squeeze(-1)
        joints_3d = joint_coords.view(B, 17, 3)  # (B, 17, 3)
        
        # 可见性
        visibility_heatmaps = self.visibility_head(x)  # (B, 17, H, W)
        visibility = F.adaptive_avg_pool2d(visibility_heatmaps, 1).squeeze(-1).squeeze(-1)
        
        # 姿态分类
        pose_class = self.pose_classifier(x)
        
        return joints_3d, visibility, pose_class
    
    def detect_oop(self, joints_3d, visibility, thresholds):
        """检测 Out-of-Position 状态
        
        Args:
            joints_3d: 3D 关节坐标 (米), shape=(B, 17, 3)
            visibility: 可见性, shape=(B, 17)
            thresholds: OOP 阈值 dict
                - forward_lean: 向前倾斜阈值 (米)
                - lateral_lean: 侧向倾斜阈值 (米)
                - head_proximity: 头部与仪表盘距离阈值 (米)
        
        Returns:
            oop_status: dict
                - is_oop: bool
                - oop_type: str (forward, lateral, head, etc.)
                - severity: float (0-1)
        """
        # 提取关键关节
        nose = joints_3d[:, 0, :]      # 鼻子
        l_shoulder = joints_3d[:, 5, :] # 左肩
        r_shoulder = joints_3d[:, 6, :] # 右肩
        l_hip = joints_3d[:, 11, :]    # 左髋
        r_hip = joints_3d[:, 12, :]    # 右髋
        
        # 计算身体中心
        body_center = (l_hip + r_hip) / 2  # 髋部中心
        
        # 1. 向前倾斜检测
        forward_lean = nose[:, 2] - body_center[:, 2]  # Z 轴（前后）
        is_forward_oop = forward_lean > thresholds['forward_lean']
        
        # 2. 侧向倾斜检测
        shoulder_center = (l_shoulder + r_shoulder) / 2
        lateral_offset = shoulder_center[:, 0] - body_center[:, 0]  # X 轴（左右）
        is_lateral_oop = torch.abs(lateral_offset) > thresholds['lateral_lean']
        
        # 3. 头部距离检测
        head_proximity = nose[:, 2]  # 假设仪表盘在 Z=0 位置
        is_head_oop = head_proximity < thresholds['head_proximity']
        
        # 综合判断
        is_oop = is_forward_oop | is_lateral_oop | is_head_oop
        
        # 确定 OOP 类型
        oop_type = "normal"
        if is_forward_oop.any():
            oop_type = "forward_lean"
        elif is_lateral_oop.any():
            oop_type = "lateral_lean"
        elif is_head_oop.any():
            oop_type = "head_proximity"
        
        return {
            'is_oop': is_oop,
            'oop_type': oop_type,
            'severity': torch.max(torch.stack([
                forward_lean / thresholds['forward_lean'],
                torch.abs(lateral_offset) / thresholds['lateral_lean'],
                1 - head_proximity / thresholds['head_proximity']
            ]), dim=0)[0]
        }

---

实验结果

1. 3D 关键点精度

方法	平均误差（cm）	中位误差（cm）	可见关节	遮挡关节
RGB-only	15.2	12.3	8.5	28.7
Depth-only	11.8	9.5	10.2	14.3
本文方法（融合）	8.6	7.1	6.3	9.8

关键发现：融合方法在遮挡场景下误差降低 65%

2. OOP 检测性能

OOP 类型	召回率	精确率	F1 分数
向前倾斜	94.2%	91.5%	92.8%
侧向倾斜	89.7%	93.1%	91.4%
头部异常	92.3%	88.9%	90.6%
综合	92.1%	91.2%	91.6%

3. 边缘部署性能

平台	模型大小	推理时间	帧率	功耗
Qualcomm QCS8255	15.2 MB	38 ms	26 fps	1.8 W
TI TDA4VM	15.2 MB	42 ms	24 fps	2.1 W
NVIDIA Jetson Orin	15.2 MB	12 ms	83 fps	5.2 W

---

IMS 开发启示

1. 传感器选型

# 推荐传感器配置
RECOMMENDED_SENSORS = {
    'depth_camera': {
        'model': 'Orbbec Astra Mini',
        'resolution': '640×480',
        'fps': 30,
        'range': '0.4-2.0m',
        'cost': 150,  # USD
        'interface': 'USB 3.0'
    },
    'ir_camera': {
        'model': 'OV2311 (RGB-IR)',
        'resolution': '1600×1200',
        'fps': 25,
        'wavelength': '940nm',
        'cost': 80,
        'interface': 'MIPI CSI'
    },
    'fusion_board': {
        'model': 'Qualcomm QCS8255',
        'cpu': 'Kryo 295',
        'npu': 'Hexagon 695',
        'ram': '8GB LPDDR5',
        'cost': 250,
        'notes': '支持 INT8 量化'
    }
}

# 总成本估算
total_cost = sum([s['cost'] for s in RECOMMENDED_SENSORS.values()])
print(f"单套硬件成本: ${total_cost} USD")
print(f"量产预计成本: ${total_cost * 0.6:.0f} USD（60%折扣）")

2. 部署流程

graph LR
    A[训练模型<br/>PyTorch FP32] --> B[导出 ONNX]
    B --> C[INT8 量化<br/>Calibration Dataset]
    C --> D[编译 QNN<br/>Qualcomm AI Hub]
    D --> E[部署测试<br/>QCS8255]
    E --> F{精度验证<br/>< 10cm 误差?}
    F -->|Yes| G[量产部署]
    F -->|No| H[微调量化参数]
    H --> C

3. OOP 检测场景清单

# Euro NCAP 2026 OOP 检测场景
OOP_TEST_SCENARIOS = [
    {
        'id': 'OOP-01',
        'name': '驾驶员向前倾斜',
        'description': '驾驶员身体前倾超过安全距离（距座椅靠背 > 30cm）',
        'detection_requirements': {
            'threshold': 0.3,  # 米
            'latency': '< 500ms',
            'accuracy': '> 90%'
        }
    },
    {
        'id': 'OOP-02',
        'name': '驾驶员侧向倾斜',
        'description': '驾驶员身体侧倾超出座椅边缘（> 15cm）',
        'detection_requirements': {
            'threshold': 0.15,
            'latency': '< 500ms',
            'accuracy': '> 88%'
        }
    },
    {
        'id': 'OOP-03',
        'name': '头部异常位置（仪表盘）',
        'description': '头部距离仪表盘 < 20cm',
        'detection_requirements': {
            'threshold': 0.2,
            'latency': '< 300ms',
            'accuracy': '> 92%'
        }
    },
    {
        'id': 'OOP-04',
        'name': '头部异常位置（侧窗）',
        'description': '头部距离侧窗 < 10cm',
        'detection_requirements': {
            'threshold': 0.1,
            'latency': '< 300ms',
            'accuracy': '> 90%'
        }
    },
    {
        'id': 'OOP-05',
        'name': '儿童座椅异常姿态',
        'description': '儿童在安全座椅中的异常姿态（躺卧、侧倾）',
        'detection_requirements': {
            'threshold': 'dynamic',
            'latency': '< 1s',
            'accuracy': '> 85%'
        }
    }
]

---

代码复现

完整推理脚本

"""
车辆乘员 3D 姿态估计推理脚本

论文：Three-Dimensional Posture Estimation of Vehicle Occupants Using Depth and Infrared Images
期刊：Sensors 2024, 24(17), 5530

复现说明：
- 输入：深度图 + 红外图（对齐）
- 输出：17 个身体关节的 3D 坐标 + OOP 检测结果
"""

import cv2
import numpy as np
import torch
from typing import Tuple, Dict

class Occupant3DPoseEstimator:
    """乘员 3D 姿态估计器"""
    
    def __init__(self, model_path: str, device: str = 'cuda'):
        """
        Args:
            model_path: 模型权重路径
            device: 推理设备 ('cuda' or 'cpu')
        """
        self.device = device
        
        # 加载模型
        self.model = self._load_model(model_path)
        self.model.eval()
        
        # OOP 阈值（默认值，可配置）
        self.oop_thresholds = {
            'forward_lean': 0.3,      # 米
            'lateral_lean': 0.15,     # 米
            'head_proximity': 0.2     # 米
        }
        
    def _load_model(self, model_path):
        """加载模型（简化实现）"""
        # 实际实现需要定义完整网络结构
        # 这里返回一个占位符
        return torch.jit.load(model_path, map_location=self.device)
    
    def preprocess(self, depth_img: np.ndarray, ir_img: np.ndarray) -> torch.Tensor:
        """预处理
        
        Args:
            depth_img: 深度图, shape=(H, W), dtype=uint16, 单位毫米
            ir_img: 红外图, shape=(H, W), dtype=uint8
            
        Returns:
            inputs: 模型输入张量, shape=(1, 2, H, W)
        """
        # 归一化
        depth_normalized = depth_img.astype(np.float32) / 5000.0  # 0-5m -> 0-1
        ir_normalized = ir_img.astype(np.float32) / 255.0
        
        # 调整尺寸
        target_size = (256, 256)
        depth_resized = cv2.resize(depth_normalized, target_size)
        ir_resized = cv2.resize(ir_normalized, target_size)
        
        # 转换为张量
        depth_tensor = torch.from_numpy(depth_resized).unsqueeze(0).unsqueeze(0)
        ir_tensor = torch.from_numpy(ir_resized).unsqueeze(0).unsqueeze(0)
        
        # 拼接
        inputs = torch.cat([depth_tensor, ir_tensor], dim=1)  # (1, 2, H, W)
        
        return inputs.to(self.device)
    
    def postprocess(self, 
                    joints_3d: torch.Tensor, 
                    visibility: torch.Tensor,
                    pose_class: torch.Tensor,
                    depth_img: np.ndarray) -> Dict:
        """后处理
        
        Args:
            joints_3d: 3D 关节坐标, shape=(1, 17, 3)
            visibility: 可见性, shape=(1, 17)
            pose_class: 姿态分类, shape=(1, 3)
            depth_img: 原始深度图（用于坐标反归一化）
            
        Returns:
            result: 检测结果 dict
        """
        # 转换为 numpy
        joints_3d = joints_3d[0].cpu().numpy()  # (17, 3)
        visibility = visibility[0].cpu().numpy()  # (17,)
        pose_class = pose_class[0].cpu().numpy()  # (3,)
        
        # 坐标反归一化（从 0-1 映射到实际距离）
        scale_factor = 5.0  # 米
        joints_3d_meters = joints_3d * scale_factor
        
        # 姿态分类
        class_names = ['normal', 'oop', 'unknown']
        predicted_class = class_names[np.argmax(pose_class)]
        
        # OOP 检测
        oop_status = self._detect_oop(joints_3d_meters, visibility)
        
        return {
            'joints_3d': joints_3d_meters,
            'visibility': visibility,
            'pose_class': predicted_class,
            'confidence': float(np.max(pose_class)),
            'oop_status': oop_status
        }
    
    def _detect_oop(self, joints_3d, visibility) -> Dict:
        """检测 OOP 状态（简化实现）"""
        # 提取关键关节
        nose = joints_3d[0]       # 鼻子
        l_shoulder = joints_3d[5] # 左肩
        r_shoulder = joints_3d[6] # 右肩
        l_hip = joints_3d[11]     # 左髋
        r_hip = joints_3d[12]     # 右髋
        
        # 计算中心
        body_center_z = (l_hip[2] + r_hip[2]) / 2
        
        # 向前倾斜
        forward_lean = nose[2] - body_center_z
        is_forward_oop = forward_lean > self.oop_thresholds['forward_lean']
        
        # 侧向倾斜
        shoulder_center_x = (l_shoulder[0] + r_shoulder[0]) / 2
        body_center_x = (l_hip[0] + r_hip[0]) / 2
        lateral_offset = abs(shoulder_center_x - body_center_x)
        is_lateral_oop = lateral_offset > self.oop_thresholds['lateral_lean']
        
        # 头部距离
        head_proximity = nose[2]
        is_head_oop = head_proximity < self.oop_thresholds['head_proximity']
        
        # 综合判断
        is_oop = is_forward_oop or is_lateral_oop or is_head_oop
        
        return {
            'is_oop': is_oop,
            'forward_lean': is_forward_oop,
            'lateral_lean': is_lateral_oop,
            'head_proximity': is_head_oop,
            'severity': max(
                forward_lean / self.oop_thresholds['forward_lean'],
                lateral_offset / self.oop_thresholds['lateral_lean'],
                1 - head_proximity / self.oop_thresholds['head_proximity']
            )
        }
    
    def infer(self, depth_img: np.ndarray, ir_img: np.ndarray) -> Dict:
        """推理
        
        Args:
            depth_img: 深度图, shape=(H, W), dtype=uint16
            ir_img: 红外图, shape=(H, W), dtype=uint8
            
        Returns:
            result: 检测结果
        """
        # 预处理
        inputs = self.preprocess(depth_img, ir_img)
        
        # 推理
        with torch.no_grad():
            joints_3d, visibility, pose_class = self.model(inputs)
        
        # 后处理
        result = self.postprocess(joints_3d, visibility, pose_class, depth_img)
        
        return result


# 实际测试
if __name__ == "__main__":
    # 模拟数据
    np.random.seed(42)
    
    # 生成模拟深度图
    depth_img = np.random.randint(500, 2000, (480, 640), dtype=np.uint16)
    
    # 生成模拟红外图
    ir_img = np.random.randint(0, 255, (480, 640), dtype=np.uint8)
    
    # 推理（需要实际模型）
    # estimator = Occupant3DPoseEstimator('model.pt')
    # result = estimator.infer(depth_img, ir_img)
    
    # 输出示例
    example_result = {
        'joints_3d': np.random.randn(17, 3) * 0.5,  # 米
        'visibility': np.random.rand(17),
        'pose_class': 'normal',
        'confidence': 0.92,
        'oop_status': {
            'is_oop': False,
            'forward_lean': False,
            'lateral_lean': False,
            'head_proximity': False,
            'severity': 0.3
        }
    }
    
    print("推理结果示例：")
    print(f"姿态分类: {example_result['pose_class']}")
    print(f"置信度: {example_result['confidence']:.2f}")
    print(f"OOP 状态: {example_result['oop_status']['is_oop']}")
    print(f"鼻尖位置 (米): {example_result['joints_3d'][0]}")

---

参考文献

1. Tambwekar, A., et al. (2024). Three-Dimensional Posture Estimation of Vehicle Occupants Using Depth and Infrared Images. Sensors, 24(17), 5530.

2. Euro NCAP (2025). Occupant Monitoring System (OMS) Test and Assessment Protocol v1.0.

---

总结

本研究提出的 深度+红外融合 3D 姿态估计方法，在遮挡、光照变化场景下显著优于传统 RGB 方法，为 Euro NCAP 2026 OOP 检测要求提供了可行的技术方案。

IMS 开发启示：

1. 传感器选型：深度摄像头 + 红外摄像头组合，成本约 $230/套
2. 部署平台：Qualcomm QCS8255 可实现 26fps 实时推理
3. 精度目标：中位误差 < 10cm，满足 OOP 检测需求
4. 量产时间线：2026 年中可完成系统集成，配合 Euro NCAP 2026 认证

---

论文来源：Sensors 2024 | DOI: 10.3390/s24175530 | IMS 研究笔记