Dapalm | IMS技术前沿

Deep In Sight + Murata 60GHz融合感知：rPPG生命体征+雷达多模态——InCabin 2026

来源信息

• 事件： InCabin Europe 2026 Press Conference
• 合作方： Deep In Sight (韩国AI公司) + Murata (村田制作所)
• 技术： Vision AI + 60GHz雷达 + rPPG
• 链接： https://incabin.com/blog/incabin-europe-2026-press-conference-recap/

核心方案

Deep In Sight与Murata联合展示多模态座舱感知概念验证：将基于视觉的DMS/OMS软件与60GHz车内雷达融合，实现比单一模态更可靠的乘员状态检测。

graph LR
    subgraph 视觉模态
        A[车内相机] --> B[Vision AI]
        B --> C[DMS: 疲劳/分心/视线]
        B --> D[OMS: 乘员分类/姿态]
        B --> E[rPPG: 心率估计]
    end
    subgraph 雷达模态
        F[60GHz Murata雷达] --> G[点云+微多普勒]
        G --> H[CPD: 隐藏儿童检测]
        G --> I[生命体征: 呼吸/心跳]
        G --> J[入侵检测]
    end
    C --> K[融合层]
    D --> K
    E --> K
    H --> K
    I --> K
    J --> K
    K --> L[统一乘员状态]

三个关键应用

1. 相机盲区补充：隐藏儿童检测

Deep In Sight CTO Inchan Ji指出：”相机有明确局限性，雷达可以克服这些挑战并实现新应用。”

import numpy as np

class CameraRadarFusion:
    """
    Deep In Sight + Murata 多模态融合方案
    
    相机: 乘员可见区域检测+分类
    雷达: 相机盲区生命体征检测
    """
    
    def __init__(self):
        self.camera_confidence = 0.0
        self.radar_confidence = 0.0
        self.fusion_result = None
    
    def camera_detect(self, image_features: dict) -> dict:
        """相机检测结果"""
        return {
            'occupants_visible': image_features.get('detected_persons', []),
            'child_visible': image_features.get('child_detected', False),
            'confidence': image_features.get('confidence', 0.9),
            'blind_spot': image_features.get('blind_spot', False)
        }
    
    def radar_detect(self, radar_data: dict) -> dict:
        """雷达检测结果"""
        return {
            'presence_detected': radar_data.get('presence', False),
            'breathing_rate': radar_data.get('breathing_rate', 0),
            'heart_rate': radar_data.get('heart_rate', 0),
            'location': radar_data.get('location', 'unknown'),
            'confidence': radar_data.get('confidence', 0.85)
        }
    
    def fuse(self, cam_result: dict, rad_result: dict) -> dict:
        """多模态融合"""
        # 情况1: 相机可见 + 雷达确认
        if cam_result['occupants_visible'] and rad_result['presence_detected']:
            return {
                'status': 'confirmed_occupant',
                'category': cam_result.get('occupants_visible', ['adult']),
                'breathing': rad_result['breathing_rate'] > 0,
                'confidence': 0.98
            }
        
        # 情况2: 相机盲区 + 雷达检测到生命体
        if cam_result['blind_spot'] and rad_result['presence_detected']:
            return {
                'status': 'hidden_occupant_alert',
                'category': 'potential_child',
                'breathing': rad_result['breathing_rate'] > 0,
                'heart_rate': rad_result['heart_rate'],
                'confidence': 0.90,
                'action': 'CPD_ALERT'
            }
        
        # 情况3: 相机可见 + 雷达无检测（误报检查）
        if cam_result['occupants_visible'] and not rad_result['presence_detected']:
            return {
                'status': 'camera_only_detection',
                'confidence': 0.60,
                'note': 'Possible false positive - verify'
            }
        
        # 情况4: 两者均无检测
        return {
            'status': 'empty_cabin',
            'confidence': 0.99
        }


# ==================== rPPG 生命体征 ====================

class RPPGVitalSigns:
    """
    rPPG (remote photoplethysmography) 远程光电容积脉搏波
    
    Deep In Sight展示: 使用标准车内相机图像
    估计心率和生命体征，无需物理接触
    
    原理: 皮肤颜色微小变化反映血液脉动
    """
    
    def __init__(self, fs: float = 30.0):
        self.fs = fs  # 采样率(相机帧率)
        
    def extract_skin_roi(self, face_landmarks: np.ndarray) -> np.ndarray:
        """从面部landmarks提取皮肤ROI"""
        # 额头区域（最常用于rPPG）
        forehead = face_landmarks[19:25]  # 假设19-24是额头上界标
        # 脸颊区域
        cheek_l = face_landmarks[31:36]
        cheek_r = face_landmarks[37:42]
        
        # 合并ROI
        roi_points = np.vstack([forehead, cheek_l, cheek_r])
        return roi_points
    
    def extract_pulse_signal(self, frames: np.ndarray, 
                             roi_mask: np.ndarray) -> np.ndarray:
        """
        从视频帧序列提取脉搏信号
        
        Args:
            frames: (T, H, W, 3) RGB视频帧
            roi_mask: (H, W) 皮肤区域mask
            
        Returns:
            pulse_signal: (T,) 脉搏信号
        """
        T = frames.shape[0]
        pulse = np.zeros(T)
        
        for t in range(T):
            frame = frames[t]
            # 提取皮肤区域像素
            skin_pixels = frame[roi_mask > 0]
            
            if len(skin_pixels) > 0:
                # Green通道（血液吸收峰）
                green_mean = np.mean(skin_pixels[:, 1])
                pulse[t] = green_mean
        
        # 去趋势
        from scipy import signal as sig
        pulse = sig.detrend(pulse, type='linear')
        
        # 带通滤波 0.7-3.0 Hz (42-180 bpm)
        nyq = self.fs / 2
        b, a = sig.butter(4, [0.7/nyq, 3.0/nyq], btype='band')
        pulse_filtered = sig.filtfilt(b, a, pulse)
        
        return pulse_filtered
    
    def estimate_heart_rate(self, pulse_signal: np.ndarray) -> float:
        """从脉搏信号估计心率"""
        from scipy import signal as sig
        
        # FFT
        freqs = np.fft.rfftfreq(len(pulse_signal), d=1/self.fs)
        fft_mag = np.abs(np.fft.rfft(pulse_signal))
        
        # 在0.7-3.0 Hz范围内找峰值
        mask = (freqs >= 0.7) & (freqs <= 3.0)
        peak_idx = np.argmax(fft_mag[mask])
        peak_freq = freqs[mask][peak_idx]
        
        hr_bpm = peak_freq * 60
        return hr_bpm


# ==================== 低功耗入侵检测 ====================

class LowPowerIntrusionDetection:
    """
    低功耗入侵检测模式
    
    Deep In Sight方案: 雷达持续运行检测入侵
    检测到后唤醒相机确认
    
    功耗: 雷达~0.5W vs 相机~2-3W
    """
    
    def __init__(self):
        self.radar_active = True
        self.camera_active = False
        self.detection_threshold = 0.6
        
    def run_radar_scan(self) -> dict:
        """雷达持续扫描"""
        # 模拟: 检测到有人接近车辆
        return {
            'presence': True,
            'distance': 0.5,  # 0.5m
            'movement': True,
            'confidence': 0.75
        }
    
    def process(self, radar_result: dict) -> dict:
        """处理逻辑"""
        if radar_result['confidence'] > self.detection_threshold:
            # 唤醒相机
            self.camera_active = True
            return {
                'action': 'wake_camera',
                'reason': 'radar_detected_presence',
                'radar_confidence': radar_result['confidence']
            }
        return {'action': 'continue_radar', 'reason': 'no_detection'}


# ==================== 测试 ====================
if __name__ == "__main__":
    print("=" * 60)
    print("Deep In Sight + Murata 多模态融合感知")
    print("InCabin Europe 2026")
    print("=" * 60)
    
    # 融合测试
    fusion = CameraRadarFusion()
    
    scenarios = [
        ("正常驾驶", 
         {'detected_persons': ['adult_driver'], 'confidence': 0.95, 'blind_spot': False},
         {'presence': True, 'breathing_rate': 16, 'heart_rate': 72, 'confidence': 0.90}),
        ("隐藏儿童(相机盲区)",
         {'detected_persons': [], 'confidence': 0.3, 'blind_spot': True},
         {'presence': True, 'breathing_rate': 28, 'heart_rate': 110, 'confidence': 0.88}),
        ("空车",
         {'detected_persons': [], 'confidence': 0.95, 'blind_spot': False},
         {'presence': False, 'breathing_rate': 0, 'heart_rate': 0, 'confidence': 0.95}),
        ("相机误报",
         {'detected_persons': ['unknown'], 'confidence': 0.4, 'blind_spot': False},
         {'presence': False, 'breathing_rate': 0, 'heart_rate': 0, 'confidence': 0.95}),
    ]
    
    print(f"\n{'场景':<20} {'融合结果':<25} {'置信度':<10} {'动作':<15}")
    print("-" * 70)
    for name, cam_data, rad_data in scenarios:
        cam_result = fusion.camera_detect(cam_data)
        rad_result = fusion.radar_detect(rad_data)
        result = fusion.fuse(cam_result, rad_result)
        
        print(f"{name:<20} {result['status']:<25} "
              f"{result.get('confidence', 0):.0%}     "
              f"{result.get('action', 'none'):<15}")
    
    # rPPG测试
    print(f"\n{'='*60}")
    print("rPPG 生命体征估计")
    print(f"{'='*60}")
    
    rppg = RPPGVitalSigns(fs=30)
    
    # 模拟30秒视频帧
    np.random.seed(42)
    T = 900  # 30s @ 30fps
    frames = np.random.randint(0, 255, (T, 64, 64, 3), dtype=np.uint8)
    
    # 添加模拟脉搏信号到皮肤区域
    t = np.arange(T) / 30.0
    pulse_freq = 1.2  # 72 bpm
    for i in range(T):
        frames[i, 20:40, 20:40, 1] = 128 + 10 * np.sin(2 * np.pi * pulse_freq * t[i])
    
    roi_mask = np.zeros((64, 64))
    roi_mask[20:40, 20:40] = 1
    
    pulse = rppg.extract_pulse_signal(frames, roi_mask)
    hr = rppg.estimate_heart_rate(pulse)
    
    print(f"  视频时长: {T/30:.1f}s")
    print(f"  估计心率: {hr:.0f} bpm")
    print(f"  真实值: 72 bpm")
    print(f"  误差: {abs(hr-72):.0f} bpm")
    
    # 功耗分析
    print(f"\n{'='*60}")
    print("低功耗入侵检测功耗分析")
    print(f"{'='*60}")
    
    intrusion = LowPowerIntrusionDetection()
    result = intrusion.run_radar_scan()
    action = intrusion.process(result)
    
    print(f"  雷达持续模式: ~0.5W (24h = 12Wh)")
    print(f"  相机持续模式: ~2.5W (24h = 60Wh)")
    print(f"  雷达触发相机: 平均 ~0.8W (节省68%)")
    print(f"  当前动作: {action['action']}")
    
    print(f"\n关键洞察 (Deep In Sight CTO Inchan Ji):")
    print(f"  '相机有明确局限性，但雷达可以克服这些挑战并实现新应用'")
    print(f"  '雷达系统正变得比相机系统更经济'")
    print(f"  '这为OEM和Tier1创造了添加新感知能力的真实机会'")

IMS应用启示

1. rPPG与DMS相机复用

功能	传感器	附加成本	输出	精度
DMS疲劳	NIR相机	$0	PERCLOS+眨眼	高
rPPG心率	同上NIR相机	$0	心率bpm	±3bpm
呼吸率	同上	$0	次/分钟	±2次
压力估计	同上	$0	HRV指标	中
酒驾筛查	同上+HRV	$0	风险评分	研究中

2. 雷达+相机融合架构

graph TD
    subgraph 停车态
        R1[雷达: 持续 0.5W] --> R2{检测到入侵?}
        R2 -->|是| C1[唤醒相机]
        R2 -->|否| R1
        C1 --> C2[相机确认]
        C2 --> C3{生命体征?}
        C3 -->|是| CPD[CPD告警]
        C3 -->|否| ALARM[入侵告警]
    end
    subgraph 行驶态
        CAM[相机: DMS/OMS] --> F1[视觉分类]
        RAD[雷达: 60GHz] --> F2[生命体征+微多普勒]
        F1 --> FUSE[融合层]
        F2 --> FUSE
        FUSE --> OUT[统一乘员状态]
    end

3. 成本-功能矩阵

配置	传感器	成本	功能覆盖	Euro NCAP
最小	NIR相机	$10	DMS+OMS+rPPG	✅ 基本
标准	NIR相机+UWB	$16	上述+CPD	✅ 完整
高配	NIR相机+60GHz	$25	上述+全舱雷达感知	✅ 完整+加分
全融合	NIR+60GHz+织物	$28	上述+重量+姿态	✅ 最佳

总结

Deep In Sight+Murata的方案展示了”相机+雷达+rPPG”三合一的终极融合策略。关键洞察是：相机解决”看到什么”，雷达解决”看不到什么”，rPPG解决”感受什么”。对IMS：rPPG是零成本扩展DMS功能的关键——复用现有NIR相机即可获得心率/呼吸率/压力指标，无需额外传感器。