manD 1.0 多模态驾驶员监测数据集：Scientific Data 2024 论文解读与使用指南

一、数据集概述

1.1 论文信息

标题： A multimodal driver monitoring benchmark dataset for driver modeling in assisted driving automation

期刊： Scientific Data (Nature 子刊), 2024

DOI： 10.1038/s41597-024-03137-y

数据仓库： Harvard Dataverse - manD 1.0

1.2 数据集特点

特性	数值
参与者数量	50 人（39 人有效数据，11 人晕动症）
驾驶场景	5 个情绪诱导场景 + 1 个熟悉化场景
传感器	7 种（EEG、ECG、眼动、摄像头、PPG、EDA、压力传感器）
采样率	统一 256 Hz（视频 30 FPS）
数据格式	.txt、.mp4、.png
总时长	约 200+ 小时

1.3 传感器配置

# manD 1.0 传感器配置

sensor_configuration = {
    "Intel RealSense D435": {
        "type": "RGB-Depth Camera",
        "resolution": "640×480 (原始), 200×200 (裁剪)",
        "fps": 30,
        "position": "显示屏上方 1.5m",
        "data": "面部视频（裁剪）"
    },
    
    "SmartEye Aurora": {
        "type": "Eye Tracker",
        "sampling_rate": "120 Hz",
        "position": "仪表盘上方 0.7m",
        "data": ["PERCLOS", "瞳孔直径", "注视点", "眨眼", "头部姿态"]
    },
    
    "Empatica E4": {
        "type": "Wearable Wristband",
        "sensors": ["PPG (64 Hz)", "EDA (4 Hz)", "加速度 (32 Hz)", "温度 (4 Hz)"],
        "data": ["心率", "HRV", "皮肤电导", "皮肤温度"]
    },
    
    "BIOPAC B-ALERT X10": {
        "type": "EEG/ECG Sensor",
        "eeg_channels": 9,  # Poz, Fz, Cz, C3, C4, F3, F4, P3, P4
        "ecg_channels": 1,
        "sampling_rate": "256 Hz",
        "data": ["脑电", "心电", "心率"]
    },
    
    "BodiTrak2 Pro": {
        "type": "Seat Pressure Sensor Mat",
        "grid": "32×32 (1024 传感器)",
        "area": "0.45×0.45 m",
        "position": ["座椅", "靠背"],
        "sampling_rate": "15 Hz (原始), 256 Hz (同步)"
    }
}

---

二、驾驶场景设计

2.1 情绪诱导场景

数据集包含 5 个情绪诱导场景：

场景	诱导情绪	视频片段	背景音乐	天气	NDRT 难度
No Emotion	中性	无	无	中午	1-back → 2-back
Anger	愤怒	Seven (6’7”)	无	中午	2-back
Surprise	惊讶	无	无	雪天→晴天	2-back → 3-back
Sadness	悲伤	City of Angels (4’25”)	Adagio in G Minor	雨天	2-back → 4-back
Fear	恐惧	The Shining (4’25”)	Night on Bald Mountain	黑暗	2-back → 5-back

2.2 场景结构

每个场景包含以下事件序列：

# 场景事件序列示例（Sadness 场景）

sadness_scenario_events = [
    {"event": "情绪视频播放", "duration": "4'25\"", "purpose": "诱导悲伤情绪"},
    {"event": "手动驾驶开始", "duration": "变长", "purpose": "建立基线"},
    {"event": "注意力对象", "duration": "~30s", "objects": ["行人", "对向来车", "自行车"]},
    {"event": "车跟随 (L2)", "duration": "~60s", "purpose": "低负荷监控"},
    {"event": "自动驾驶 (L3)", "duration": "~90s", "purpose": "高负荷 NDRT"},
    {"event": "N-back 游戏", "duration": "30s/轮", "difficulty": "2 → 4"},
    {"event": "听觉数字广度任务", "duration": "~60s", "difficulty": "4"},
    {"event": "朗读书籍", "duration": "~30s", "purpose": "手动驾驶分心"},
    {"event": "TOR（马群穿越）", "duration": "≤10s", "type": "简单 TOR"},
    {"event": "情绪中性段", "duration": "~60s", "purpose": "情绪恢复"},
]

---

三、数据下载与结构

3.1 下载地址

Harvard Dataverse： https://doi.org/10.7910/DVN/XXXXX（实际 DOI 见论文）

3.2 数据结构

manD 1.0/
├── AvailableData.xlsx           # 数据可用性报告
├── DataInfoSheet.xlsx           # 数据字段说明
├── DESResults.xlsx              # DES 问卷结果（Z-Score）
├── ParticipantsCharacteristics.xlsx  # 参与者特征
├── EEG-BIDS.7z                  # EEG 数据（BIDS 格式）
├── DataExtraction.py            # 数据提取脚本
├── withoutMS/                   # 无晕动症参与者
│   ├── P1.7z/
│   │   ├── Familiarization/
│   │   │   ├── EnvironmentState/
│   │   │   │   └── EnvironmentState.txt
│   │   │   ├── VehicleState/
│   │   │   │   └── VehicleState.txt
│   │   │   └── DriverState/
│   │   │       ├── Face_cropped.mp4
│   │   │       ├── ActivityLabels.txt
│   │   │       ├── EyeTracking.txt
│   │   │       ├── PhysiologicalData/
│   │   │       │   ├── PPG_EDA.txt
│   │   │       │   └── EEG_ECG.txt
│   │   │       ├── SeatPressureSensor/
│   │   │       │   ├── SeatPressureDistribution.txt
│   │   │       │   └── Media/
│   │   │       └── NDRT/
│   │   │           ├── NDRT_nback.txt
│   │   │           ├── NDRT_Digit_Span.txt
│   │   │           └── NDRT_SubwaySurfers.txt
│   │   ├── NoEmotion/
│   │   ├── Anger/
│   │   ├── Surprise/
│   │   ├── Sadness/
│   │   └── Fear/
│   ├── P2.7z/
│   └── ...
└── withMS/                      # 有晕动症参与者
    └── ...

---

四、数据加载代码

4.1 Python 数据加载器

"""
manD 1.0 数据加载器

论文：A multimodal driver monitoring benchmark dataset for driver modeling in assisted driving automation
期刊：Scientific Data (Nature), 2024
数据仓库：Harvard Dataverse
"""

import numpy as np
import pandas as pd
import cv2
from pathlib import Path
from typing import Dict, List, Optional, Tuple
from dataclasses import dataclass
import matplotlib.pyplot as plt

@dataclass
class DriverStateData:
    """驾驶员状态数据"""
    timestamp: np.ndarray
    
    # 眼动数据
    perclos: np.ndarray          # PERCLOS (%)
    pupil_diameter: np.ndarray   # 瞳孔直径 (mm)
    gaze_x: np.ndarray           # 注视点 X
    gaze_y: np.ndarray           # 注视点 Y
    head_position: np.ndarray    # 头部位置 (3D)
    head_rotation: np.ndarray    # 头部旋转 (3D)
    
    # 生理数据 - Empatica E4
    hr_empatica: np.ndarray      # 心率 (BPM) - PPG
    hrv: np.ndarray              # 心率变异
    eda: np.ndarray              # 皮肤电活动 (µS)
    scl: np.ndarray              # 皮肤电导水平
    scr: np.ndarray              # 皮肤电导响应
    temperature: np.ndarray      # 皮肤温度 (°C)
    
    # 生理数据 - BIOPAC
    eeg: np.ndarray              # EEG (9 channels)
    ecg: np.ndarray              # ECG
    hr_biopac: np.ndarray        # 心率 - ECG
    
    # 座椅压力
    seat_pressure: np.ndarray    # 座椅压力分布 (32×32)
    back_pressure: np.ndarray    # 靠背压力分布 (32×32)
    
    # 活动标签
    activity: np.ndarray         # 活动标签


class ManD1Dataset:
    """
    manD 1.0 数据集加载器
    
    Example:
        >>> dataset = ManD1Dataset('/path/to/manD1.0')
        >>> data = dataset.load_participant('P1', 'Sadness')
        >>> print(f"Duration: {len(data.timestamp) / 256:.1f} seconds")
    """
    
    def __init__(self, root_path: str):
        self.root = Path(root_path)
        self.sampling_rate = 256
        
        # 参与者列表
        self.participants = self._list_participants()
        
        # 数据字段说明
        self.data_info = pd.read_excel(self.root / 'DataInfoSheet.xlsx')
        
    def _list_participants(self) -> List[str]:
        """列出所有参与者"""
        without_ms = list((self.root / 'withoutMS').glob('P*.7z'))
        with_ms = list((self.root / 'withMS').glob('P*.7z'))
        return [p.stem for p in sorted(without_ms + with_ms)]
    
    def load_participant(
        self,
        participant: str,
        scenario: str,
        data_types: Optional[List[str]] = None
    ) -> DriverStateData:
        """
        加载单个参与者的单个场景数据
        
        Args:
            participant: 参与者 ID（如 'P1'）
            scenario: 场景名称（'Familiarization', 'NoEmotion', 'Anger', 'Surprise', 'Sadness', 'Fear'）
            data_types: 要加载的数据类型，None 表示全部
        
        Returns:
            DriverStateData: 统一格式的数据对象
        """
        # 确定目录
        if participant in [p.stem for p in (self.root / 'withoutMS').glob('P*.7z')]:
            base_path = self.root / 'withoutMS' / participant / scenario / 'DriverState'
        else:
            base_path = self.root / 'withMS' / participant / scenario / 'DriverState'
        
        # 加载眼动数据
        eye_data = self._load_eye_tracking(base_path / 'EyeTracking.txt')
        
        # 加载生理数据
        ppg_eda = self._load_ppg_eda(base_path / 'PhysiologicalData' / 'PPG_EDA.txt')
        eeg_ecg = self._load_eeg_ecg(base_path / 'PhysiologicalData' / 'EEG_ECG.txt')
        
        # 加载座椅压力
        pressure = self._load_pressure(base_path / 'SeatPressureSensor' / 'SeatPressureDistribution.txt')
        
        # 加载活动标签
        activity = self._load_activity(base_path / 'ActivityLabels.txt')
        
        return DriverStateData(
            timestamp=np.arange(len(eye_data['perclos'])) / self.sampling_rate,
            perclos=eye_data['perclos'],
            pupil_diameter=eye_data['pupil_diameter'],
            gaze_x=eye_data['gaze_x'],
            gaze_y=eye_data['gaze_y'],
            head_position=eye_data['head_position'],
            head_rotation=eye_data['head_rotation'],
            hr_empatica=ppg_eda['hr'],
            hrv=ppg_eda['hrv'],
            eda=ppg_eda['eda'],
            scl=ppg_eda['scl'],
            scr=ppg_eda['scr'],
            temperature=ppg_eda['temperature'],
            eeg=eeg_ecg['eeg'],
            ecg=eeg_ecg['ecg'],
            hr_biopac=eeg_ecg['hr'],
            seat_pressure=pressure['seat'],
            back_pressure=pressure['back'],
            activity=activity
        )
    
    def _load_eye_tracking(self, path: Path) -> dict:
        """加载眼动追踪数据"""
        df = pd.read_csv(path, sep='\t')
        return {
            'perclos': df['PERCLOS'].values,
            'pupil_diameter': df['PupilDiameter'].values,
            'gaze_x': df['GazeX'].values,
            'gaze_y': df['GazeY'].values,
            'head_position': df[['HeadX', 'HeadY', 'HeadZ']].values,
            'head_rotation': df[['HeadRotX', 'HeadRotY', 'HeadRotZ']].values
        }
    
    def _load_ppg_eda(self, path: Path) -> dict:
        """加载 PPG/EDA 生理数据"""
        df = pd.read_csv(path, sep='\t')
        return {
            'hr': df['Empatica/HR'].values,
            'hrv': df.get('Empatica/HRV', np.zeros(len(df))).values,
            'eda': df['Empatica/EDA'].values,
            'scl': df['Empatica/SCL'].values,
            'scr': df['Empatica/SCR'].values,
            'temperature': df['Empatica/Temperature'].values
        }
    
    def _load_eeg_ecg(self, path: Path) -> dict:
        """加载 EEG/ECG 数据"""
        df = pd.read_csv(path, sep='\t')
        eeg_channels = ['Poz', 'Fz', 'Cz', 'C3', 'C4', 'F3', 'F4', 'P3', 'P4']
        return {
            'eeg': df[eeg_channels].values,
            'ecg': df['ECG'].values,
            'hr': df['BIOPAC/HR'].values
        }
    
    def _load_pressure(self, path: Path) -> dict:
        """加载座椅压力数据"""
        df = pd.read_csv(path, sep='\t')
        
        # 座椅压力 (Seat_000 到 Seat_1023)
        seat_cols = [f'Seat_{i:03d}' for i in range(1024)]
        seat = df[seat_cols].values.reshape(-1, 32, 32)
        
        # 靠背压力 (Back_000 到 Back_1023)
        back_cols = [f'Back_{i:03d}' for i in range(1024)]
        back = df[back_cols].values.reshape(-1, 32, 32)
        
        return {'seat': seat, 'back': back}
    
    def _load_activity(self, path: Path) -> np.ndarray:
        """加载活动标签"""
        df = pd.read_csv(path, sep='\t')
        return df['Activity'].values
    
    def load_video(
        self,
        participant: str,
        scenario: str
    ) -> cv2.VideoCapture:
        """
        加载面部视频
        
        Returns:
            cv2.VideoCapture: OpenCV 视频对象
        """
        base_path = self.root / 'withoutMS' / participant / scenario / 'DriverState'
        video_path = base_path / 'Face_cropped.mp4'
        return cv2.VideoCapture(str(video_path))


# ==================== 使用示例 ====================

if __name__ == "__main__":
    # 初始化数据集
    dataset = ManD1Dataset('/path/to/manD1.0')
    
    # 加载参与者 1 的 Sadness 场景
    data = dataset.load_participant('P1', 'Sadness')
    
    print(f"数据时长: {len(data.timestamp) / 256:.1f} 秒")
    print(f"PERCLOS 范围: {data.perclos.min():.1f}% - {data.perclos.max():.1f}%")
    print(f"心率范围: {np.nanmin(data.hr_empatica):.1f} - {np.nanmax(data.hr_empatica):.1f} BPM")
    print(f"EEG 数据形状: {data.eeg.shape}")
    print(f"座椅压力数据形状: {data.seat_pressure.shape}")
    
    # 可视化 PERCLOS 和心率
    fig, axes = plt.subplots(2, 1, figsize=(12, 6), sharex=True)
    
    time = data.timestamp / 60  # 转换为分钟
    
    axes[0].plot(time, data.perclos, 'b-', linewidth=0.5)
    axes[0].set_ylabel('PERCLOS (%)')
    axes[0].set_title('驾驶员疲劳指标 (PERCLOS)')
    axes[0].grid(True, alpha=0.3)
    
    axes[1].plot(time, data.hr_empatica, 'r-', linewidth=0.5)
    axes[1].set_xlabel('时间 (分钟)')
    axes[1].set_ylabel('心率 (BPM)')
    axes[1].set_title('驾驶员心率')
    axes[1].grid(True, alpha=0.3)
    
    plt.tight_layout()
    plt.savefig('driver_state_analysis.png', dpi=150)
    print("图表已保存: driver_state_analysis.png")

---

五、数据质量评估

5.1 信号质量指标

数据类型	采样率	动态范围	分辨率	已知问题
EEG	256 Hz	±1000 µV	0.038 µV	前 24 秒数据丢弃
ECG	256 Hz	±1000 µV	0.06 µV	同上
眼动	120 Hz → 256 Hz	-	-	需重采样
PPG	64 Hz → 256 Hz	500 nW	-	需峰值检测
EDA	4 Hz → 256 Hz	-	µS	需 SCL/SCR 分解
压力	15 Hz → 256 Hz	0-100/200	10%	需填充零值帧

5.2 数据预处理建议

def preprocess_eeg(eeg_data: np.ndarray) -> np.ndarray:
    """
    EEG 预处理
    
    1. 去除前 24 秒数据（传感器稳定时间）
    2. 坏通道检测和插值
    3. 滤波（0.1-67 Hz）
    """
    # 去除前 24 秒
    eeg_data = eeg_data[256*24:]
    
    # 坏通道检测（极端值）
    mean_vals = np.mean(eeg_data, axis=0)
    bad_channels = np.abs(mean_vals) > np.mean(np.abs(mean_vals)) * 3
    
    # 插值坏通道
    if np.any(bad_channels):
        good_channels = ~bad_channels
        eeg_data[:, bad_channels] = np.mean(eeg_data[:, good_channels], axis=1, keepdims=True)
    
    return eeg_data


def preprocess_perclos(perclos: np.ndarray) -> np.ndarray:
    """
    PERCLOS 预处理
    
    1. 平滑滤波
    2. 异常值处理
    """
    from scipy.ndimage import gaussian_filter1d
    
    # 高斯平滑
    perclos_smooth = gaussian_filter1d(perclos, sigma=5)
    
    # 裁剪到合理范围
    perclos_smooth = np.clip(perclos_smooth, 0, 100)
    
    return perclos_smooth


def calculate_hrv(ppg_data: np.ndarray, fs: int = 256) -> dict:
    """
    从 PPG 计算心率变异指标
    
    Returns:
        hrv_metrics: {SDNN, RMSSD, pNN50, LF, HF, LF/HF}
    """
    from scipy.signal import find_peaks
    import neurokit2 as nk
    
    # 峰值检测
    peaks, _ = find_peaks(ppg_data, distance=fs*0.5)  # 最小 RR 间隔 0.5s
    
    # 计算 RR 间隔
    rr_intervals = np.diff(peaks) / fs * 1000  # ms
    
    # HRV 计算
    hrv = nk.hrv(rr_intervals, sampling_rate=1000)
    
    return {
        'SDNN': hrv['HRV_SDNN'].values[0],
        'RMSSD': hrv['HRV_RMSSD'].values[0],
        'pNN50': hrv['HRV_pNN50'].values[0],
        'LF': hrv['HRV_LF'].values[0],
        'HF': hrv['HRV_HF'].values[0],
        'LF_HF': hrv['HRV_LFHF'].values[0]
    }

---

六、应用场景

6.1 疲劳检测模型训练

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

def train_fatigue_model(dataset: ManD1Dataset) -> dict:
    """
    训练疲劳检测模型
    
    使用 PERCLOS + HRV + EDA 特征
    """
    features = []
    labels = []
    
    for participant in dataset.participants[:20]:  # 使用前 20 人训练
        for scenario in ['NoEmotion', 'Sadness', 'Fear']:
            data = dataset.load_participant(participant, scenario)
            
            # 特征提取（每 30 秒窗口）
            window_size = 256 * 30
            for i in range(0, len(data.perclos) - window_size, window_size):
                window = slice(i, i + window_size)
                
                feat = {
                    'perclos_mean': np.mean(data.perclos[window]),
                    'perclos_std': np.std(data.perclos[window]),
                    'hr_mean': np.nanmean(data.hr_empatica[window]),
                    'hr_std': np.nanstd(data.hr_empatica[window]),
                    'eda_mean': np.nanmean(data.eda[window]),
                    'eda_std': np.nanstd(data.eda[window]),
                }
                features.append(feat)
                
                # 标签：PERCLOS > 30% 视为疲劳
                labels.append(1 if feat['perclos_mean'] > 30 else 0)
    
    X = pd.DataFrame(features)
    y = np.array(labels)
    
    # 训练
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
    model = RandomForestClassifier(n_estimators=100)
    model.fit(X_train, y_train)
    
    # 评估
    accuracy = model.score(X_test, y_test)
    
    return {
        'model': model,
        'accuracy': accuracy,
        'feature_importance': dict(zip(X.columns, model.feature_importances_))
    }

6.2 情绪识别研究

def analyze_emotion_induction(dataset: ManD1Dataset):
    """
    分析情绪诱导效果
    
    使用 DES 问卷结果 + 生理信号
    """
    # 加载 DES 结果
    des_results = pd.read_excel(dataset.root / 'DESResults.xlsx')
    
    # 分析每个场景的情绪变化
    emotion_changes = {}
    
    for scenario in ['Anger', 'Sadness', 'Fear', 'Surprise']:
        pre_cols = [c for c in des_results.columns if f'Pre_{scenario}' in c]
        post_cols = [c for c in des_results.columns if f'Post_{scenario}' in c]
        
        pre_scores = des_results[pre_cols].mean().mean()
        post_scores = des_results[post_cols].mean().mean()
        
        emotion_changes[scenario] = post_scores - pre_scores
    
    print("情绪诱导效果（Z-Score 变化）:")
    for scenario, change in emotion_changes.items():
        print(f"  {scenario}: {change:+.3f}")

---

七、数据集价值

7.1 与其他数据集对比

数据集	时长	参与者	传感器	优势
manD 1.0	200+ h	50	7 种	多模态最全
DMD	41 h	37	RGB/IR/Depth	多视角
TICaM	-	-	RGB/Depth/IR	真实+合成
100-Driver	-	100	RGB	大规模

7.2 研究应用方向

1. 疲劳检测算法优化：多模态融合提升准确率
2. 情绪识别研究：情绪诱导效果评估
3. 认知负荷估计：n-back 难度与生理信号关联
4. 接管请求研究：TOR 响应时间预测
5. 晕动症研究：单独的数据子集

---

八、总结

核心要点

1. 最全面的多模态 DMS 数据集：7 种传感器同步采集
2. 情绪诱导设计：5 种情绪场景，标准视频/音乐刺激
3. 高质量数据：统一 256 Hz 采样，严格预处理
4. 开放获取：Harvard Dataverse 免费下载

参考文献

1. Sabry et al. (2024). “A multimodal driver monitoring benchmark dataset for driver modeling in assisted driving automation.” Scientific Data.
2. Harvard Dataverse - manD 1.0 Dataset.
3. NeuroKit2 Documentation.

---

相关文章：

• Euro NCAP 2026 认知分心检测
• 疲劳检测算法深度解析