夜间行人检测：传感器融合提升40%准确率

论文信息

  • 标题： Sensor-Fused Nighttime System for Enhanced Pedestrian Detection in ADAS and Autonomous Vehicles
  • 期刊： Sensors 2024
  • DOI： 10.3390/s24144755
  • 核心指标： 夜间行人检测准确率提升40%

核心创新

本文提出RGB+热成像融合夜间行人检测方案：

  1. 多模态融合：RGB + 热成像 + 近红外
  2. 夜间性能：黑暗环境下检测准确率提升40%
  3. 鲁棒性强：适应隧道、雨夜等极端场景
  4. 实时性：满足ADAS实时检测要求

方法详解

1. 传感器配置

flowchart LR
    subgraph 传感器层
        A1[RGB摄像头]
        A2[热成像相机]
        A3[近红外相机]
    end
    
    subgraph 融合层
        B1[特征提取]
        B2[跨模态对齐]
        B3[自适应融合]
    end
    
    subgraph 检测层
        C1[目标检测]
        C2[行人分类]
    end
    
    A1 --> B1
    A2 --> B1
    A3 --> B1
    
    B1 --> B2 --> B3 --> C1 --> C2

2. 多模态融合架构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
import torch
import torch.nn as nn
import torch.nn.functional as F

class MultiModalFusion(nn.Module):
    """
    多模态融合网络
    
    RGB + 热成像 + 近红外
    """
    
    def __init__(
        self,
        rgb_channels: int = 3,
        thermal_channels: int = 1,
        nir_channels: int = 1,
        hidden_dim: int = 256
    ):
        super().__init__()
        
        # RGB特征提取
        self.rgb_encoder = nn.Sequential(
            nn.Conv2d(rgb_channels, 64, 7, stride=2, padding=3),
            nn.ReLU(),
            nn.Conv2d(64, 128, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, hidden_dim, 3, stride=2, padding=1)
        )
        
        # 热成像特征提取
        self.thermal_encoder = nn.Sequential(
            nn.Conv2d(thermal_channels, 64, 7, stride=2, padding=3),
            nn.ReLU(),
            nn.Conv2d(64, 128, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, hidden_dim, 3, stride=2, padding=1)
        )
        
        # 近红外特征提取
        self.nir_encoder = nn.Sequential(
            nn.Conv2d(nir_channels, 64, 7, stride=2, padding=3),
            nn.ReLU(),
            nn.Conv2d(64, 128, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, hidden_dim, 3, stride=2, padding=1)
        )
        
        # 自适应融合权重
        self.fusion_gate = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),
            nn.Conv2d(hidden_dim * 3, 3, 1),
            nn.Softmax(dim=1)
        )
        
        # 融合层
        self.fusion_conv = nn.Conv2d(hidden_dim * 3, hidden_dim, 1)
    
    def forward(
        self,
        rgb: torch.Tensor,
        thermal: torch.Tensor,
        nir: torch.Tensor
    ) -> torch.Tensor:
        """
        Args:
            rgb: RGB图像, shape=(B, 3, H, W)
            thermal: 热成像, shape=(B, 1, H, W)
            nir: 近红外, shape=(B, 1, H, W)
        
        Returns:
            fused: 融合特征, shape=(B, C, H', W')
        """
        # 特征提取
        rgb_feat = self.rgb_encoder(rgb)
        thermal_feat = self.thermal_encoder(thermal)
        nir_feat = self.nir_encoder(nir)
        
        # 拼接
        concat = torch.cat([rgb_feat, thermal_feat, nir_feat], dim=1)
        
        # 计算融合权重
        weights = self.fusion_gate(concat)  # (B, 3, 1, 1)
        
        # 加权融合
        w_rgb = weights[:, 0:1, :, :]
        w_thermal = weights[:, 1:2, :, :]
        w_nir = weights[:, 2:3, :, :]
        
        weighted = torch.cat([
            rgb_feat * w_rgb,
            thermal_feat * w_thermal,
            nir_feat * w_nir
        ], dim=1)
        
        # 融合
        fused = self.fusion_conv(weighted)
        
        return fused


class PedestrianDetector(nn.Module):
    """
    行人检测器
    
    基于融合特征
    """
    
    def __init__(self, hidden_dim: int = 256):
        super().__init__()
        
        self.fusion = MultiModalFusion(hidden_dim=hidden_dim)
        
        # 检测头（简化）
        self.detector = nn.Sequential(
            nn.Conv2d(hidden_dim, 128, 3, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, 64, 3, padding=1),
            nn.ReLU()
        )
        
        # 分类头
        self.classifier = nn.Conv2d(64, 2, 1)  # 背景/行人
        
        # 回归头
        self.regressor = nn.Conv2d(64, 4, 1)  # bbox
    
    def forward(
        self,
        rgb: torch.Tensor,
        thermal: torch.Tensor,
        nir: torch.Tensor
    ) -> dict:
        """
        Args:
            rgb: RGB图像
            thermal: 热成像
            nir: 近红外
        
        Returns:
            {
                'cls': 分类logits,
                'bbox': 边界框回归
            }
        """
        # 融合特征
        fused = self.fusion(rgb, thermal, nir)
        
        # 检测
        feat = self.detector(fused)
        
        # 输出
        cls = self.classifier(feat)
        bbox = self.regressor(feat)
        
        return {
            'cls': cls,
            'bbox': bbox
        }


# 示例
if __name__ == "__main__":
    model = PedestrianDetector()
    
    # 模拟输入
    B = 2
    H, W = 480, 640
    
    rgb = torch.randn(B, 3, H, W)
    thermal = torch.randn(B, 1, H, W)
    nir = torch.randn(B, 1, H, W)
    
    # 前向传播
    result = model(rgb, thermal, nir)
    
    print(f"输入RGB: {rgb.shape}")
    print(f"输入热成像: {thermal.shape}")
    print(f"输入近红外: {nir.shape}")
    print(f"分类输出: {result['cls'].shape}")
    print(f"边界框输出: {result['bbox'].shape}")

3. 光照自适应

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
class IlluminationAdaptiveFusion(nn.Module):
    """
    光照自适应融合
    
    根据环境光照动态调整融合权重
    """
    
    def __init__(self):
        super().__init__()
        
        # 光照估计
        self.illum_estimator = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),
            nn.Flatten(),
            nn.Linear(3, 3),
            nn.Softmax(dim=-1)
        )
    
    def estimate_illumination(self, rgb: torch.Tensor) -> torch.Tensor:
        """
        估计环境光照
        
        Returns:
            weights: [rgb_weight, thermal_weight, nir_weight]
        """
        # 计算亮度
        gray = 0.299 * rgb[:, 0] + 0.587 * rgb[:, 1] + 0.114 * rgb[:, 2]
        mean_brightness = gray.mean(dim=(1, 2), keepdim=True)
        
        # 简化：根据亮度调整权重
        # 低光时增加热成像权重
        B = rgb.shape[0]
        weights = torch.zeros(B, 3, device=rgb.device)
        
        for i in range(B):
            brightness = mean_brightness[i].item()
            
            if brightness < 0.2:  # 暗光
                weights[i] = torch.tensor([0.2, 0.5, 0.3])
            elif brightness > 0.8:  # 强光
                weights[i] = torch.tensor([0.7, 0.1, 0.2])
            else:  # 正常
                weights[i] = torch.tensor([0.5, 0.2, 0.3])
        
        return weights
    
    def forward(
        self,
        rgb: torch.Tensor,
        thermal: torch.Tensor,
        nir: torch.Tensor
    ) -> torch.Tensor:
        """
        光照自适应融合
        """
        weights = self.estimate_illumination(rgb)
        
        # 加权
        fused = (
            weights[:, 0:1, None, None] * rgb.mean(dim=1, keepdim=True) +
            weights[:, 1:2, None, None] * thermal +
            weights[:, 2:3, None, None] * nir
        )
        
        return fused


# 示例
if __name__ == "__main__":
    adapter = IlluminationAdaptiveFusion()
    
    # 模拟不同光照
    rgb_dark = torch.randn(1, 3, 240, 320) * 0.1
    rgb_bright = torch.randn(1, 3, 240, 320) * 0.9 + 0.5
    thermal = torch.randn(1, 1, 240, 320)
    nir = torch.randn(1, 1, 240, 320)
    
    # 融合
    fused_dark = adapter(rgb_dark, thermal, nir)
    fused_bright = adapter(rgb_bright, thermal, nir)
    
    print("暗光融合权重:", adapter.estimate_illumination(rgb_dark))
    print("强光融合权重:", adapter.estimate_illumination(rgb_bright))

实验结果

性能对比

场景 RGB单模态 融合方法 改进
白天 92.3% 94.1% +1.8%
黄昏 78.5% 89.7% +11.2%
夜间（城郊） 54.2% 87.6% +33.4%
夜间（隧道） 61.3% 91.2% +29.9%
雨夜 48.7% 82.3% +33.6%
平均 66.9% 89.0% +22.1%

不同光照条件

光照(lux) RGB 热成像 融合 最佳模态
>1000 95% 70% 96% RGB
100-1000 88% 75% 93% RGB
10-100 62% 80% 89% 热成像
<10 35% 85% 83% 热成像

IMS开发启示

1. 夜间DMS增强

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# nighttime-dms-config.yaml
nighttime_enhancement:
  sensors:
    rgb:
      model: "OV2311"
      sensitivity: "high"
      
    thermal:
      model: "FLIR Lepton 3.5"
      resolution: "160x120"
      
    nir:
      model: "SFH 4740 + OV2311"
      wavelength: "940nm"
  
  fusion:
    method: "adaptive_weighted"
    illumination_threshold: [10, 100, 1000]  # lux
    
  weights:
    bright: [0.7, 0.1, 0.2]
    normal: [0.5, 0.2, 0.3]
    dark: [0.2, 0.5, 0.3]

2. 硬件配置

传感器 型号 分辨率 帧率 备注
RGB-IR OV2311 1600x1200 25fps RGB+近红外
热成像 FLIR Lepton 3.5 160x120 9fps 长波红外

3. 实现优先级

优先级 模块 工作量 备注
P0 热成像驱动 2周 FLIR SDK
P1 融合网络 3周 训练+部署
P1 光照自适应 1周 权重计算
P2 夜间DMS 2周 完整流程

结论

多模态传感器融合为夜间行人检测提供了可靠方案：

  1. 性能提升：夜间检测准确率提升22-40%
  2. 鲁棒性强：适应各种光照条件
  3. 实时性好：满足ADAS要求

对于IMS开发，建议：

  • P0优先集成热成像传感器
  • 实现光照自适应融合权重
  • 建立完整的夜间测试场景

参考实现： 完整代码已上传GitHub。

传感器融合
#ADAS #传感器融合 #行人检测 #夜间检测
夜间行人检测：传感器融合提升40%准确率
https://dapalm.com/2026/08/13/2026-08-14-nighttime-pedestrian-detection-fusion/
作者
Mars
发布于
2026年8月13日
许可协议