LISA：语言引导的驾驶员视线估计抗干扰框架

论文信息

• 标题：LISA: Language-guided Interference-aware Spatial-Frequency Attention for Driver Gaze Estimation
• 作者：Zhenye Yang, Ruichen Zhou, Pei Zhang, Huan Li, Jinpeng Chen（北京邮电大学）
• 会议：arXiv 2026
• 链接：https://arxiv.org/abs/2605.17287
• GitHub：https://github.com/Mason-bupt/LISA

---

核心创新

首个融合频域先验与视觉语言知识的驾驶员视线估计框架，通过语言引导的特征解耦，实现墨镜/口罩遮挡和极端光照下的鲁棒视线估计。

关键突破：

1. 频域-空域双域融合（解决光照变化）
2. 语言引导的特征解耦（解决墨镜/口罩遮挡）
3. CLIP语义嵌入分离视线特征与外观干扰
4. 遮挡场景误差降低35%

---

问题定义

现实挑战

graph TB
    A[驾驶员视线估计] --> B{挑战}
    
    B --> C[极端光照]
    B --> D[墨镜遮挡]
    B --> E[口罩遮挡]
    B --> F[传感器噪声]
    
    C --> C1[瞳孔定位失败]
    D --> D1[眼部特征丢失]
    E --> E1[下半脸不可见]
    F --> F1[特征抖动]

传统方法的局限

方法	光照鲁棒性	遮挡鲁棒性	根本原因
CNN纯空间域	❌ 差	❌ 差	像素级特征易受干扰
Transformer	🟡 中	❌ 差	Patch特征仍依赖空间分布
LISA	✅ 强	✅ 强	频域稳定+语义解耦

---

方法详解

1. 整体架构

graph TB
    subgraph 输入
        A[驾驶员图像]
    end
    
    A --> B[空间分支<br/>ResNet-18]
    A --> C[频域分支<br/>FFT变换]
    
    B --> D[空间特征]
    C --> E[频域特征<br/>幅度谱]
    
    D --> F[FAM融合模块]
    E --> F
    
    F --> G[视线意图向量]
    
    G --> H[语义解耦模块<br/>CLIP]
    H --> I[纯净化视线特征]
    
    I --> J[视线预测]

2. 核心模块实现

2.1 频域-空域融合（FAM）

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

class FrequencyAttentionModulation(nn.Module):
    """频域-空域融合模块
    
    核心思想：频域幅度谱在光照变化下保持稳定
    """
    
    def __init__(self, spatial_channels=512, freq_channels=256):
        super().__init__()
        
        # 空域特征处理
        self.spatial_conv = nn.Sequential(
            nn.Conv2d(spatial_channels, 256, 1),
            nn.BatchNorm2d(256),
            nn.ReLU()
        )
        
        # 频域特征处理
        self.freq_conv = nn.Sequential(
            nn.Conv2d(2, 64, 3, padding=1),  # 2通道：实部+虚部
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(64, 128, 3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d(1)
        )
        
        # 频域注入
        self.freq_inject = nn.Linear(128, 256)
        
        # 融合
        self.fusion = nn.Sequential(
            nn.Conv2d(256 + 256, 256, 1),
            nn.BatchNorm2d(256),
            nn.ReLU()
        )
        
        # 空间显著性门控
        self.spatial_gate = nn.Sequential(
            nn.Conv2d(256, 64, 1),
            nn.ReLU(),
            nn.Conv2d(64, 1, 1),
            nn.Sigmoid()
        )
    
    def forward(self, spatial_feat, img):
        """
        Args:
            spatial_feat: (B, C, H, W) 空域特征
            img: (B, 3, H, W) 原始图像
        
        Returns:
            fused_feat: (B, 256, H, W) 融合特征
        """
        B, C, H, W = spatial_feat.shape
        
        # 1. 空域特征处理
        spatial_feat_proc = self.spatial_conv(spatial_feat)  # (B, 256, H, W)
        
        # 2. 频域变换
        img_gray = img.mean(dim=1, keepdim=True)  # (B, 1, H, W) 灰度图
        freq = torch.fft.fft2(img_gray)  # FFT
        freq_mag = torch.abs(freq)  # 幅度谱
        freq_phase = torch.angle(freq)  # 相位谱
        
        # 归一化
        freq_mag = torch.log(freq_mag + 1e-6)  # log幅度谱
        
        # 频域特征提取
        freq_feat = self.freq_conv(torch.cat([freq_mag, freq_phase], dim=1))  # (B, 128, 1, 1)
        freq_feat = freq_feat.squeeze(-1).squeeze(-1)  # (B, 128)
        
        # 3. 频域注入
        freq_injected = self.freq_inject(freq_feat)  # (B, 256)
        freq_injected = freq_injected.unsqueeze(-1).unsqueeze(-1)  # (B, 256, 1, 1)
        
        # 4. 融合
        freq_broadcast = freq_injected.expand(-1, -1, H, W)  # (B, 256, H, W)
        concat_feat = torch.cat([spatial_feat_proc, freq_broadcast], dim=1)  # (B, 512, H, W)
        fused_feat = self.fusion(concat_feat)  # (B, 256, H, W)
        
        # 5. 空间显著性门控
        gate = self.spatial_gate(fused_feat)  # (B, 1, H, W)
        gated_feat = fused_feat * gate  # 眼部区域增强
        
        return gated_feat


class SemanticDisentanglement(nn.Module):
    """语义解耦模块
    
    使用CLIP语言嵌入分离视线特征与外观干扰
    """
    
    def __init__(self, gaze_dim=128, clip_dim=512):
        super().__init__()
        
        # 视线特征投影
        self.gaze_proj = nn.Linear(gaze_dim, clip_dim)
        
        # 干扰文本嵌入（冻结）
        self.distractor_texts = [
            "a driver wearing sunglasses",
            "a driver wearing a mask",
            "a driver with glasses",
            "a driver in dim light",
            "a driver in bright sunlight"
        ]
        
        # 注册干扰嵌入（预计算）
        self.register_buffer('distractor_embeddings', self._precompute_distractors())
        
        # 解耦层
        self.disentangle = nn.Sequential(
            nn.Linear(clip_dim, 256),
            nn.ReLU(),
            nn.Linear(256, clip_dim)
        )
    
    def _precompute_distractors(self):
        """预计算干扰文本嵌入"""
        # 实际应使用CLIP文本编码器
        # 简化：返回随机嵌入
        num_distractors = len(self.distractor_texts)
        return torch.randn(num_distractors, 512)
    
    def forward(self, gaze_feat):
        """
        Args:
            gaze_feat: (B, 128) 视线特征
        
        Returns:
            purified_gaze: (B, 512) 纯净化视线特征
        """
        # 投影到CLIP空间
        gaze_clip = self.gaze_proj(gaze_feat)  # (B, 512)
        
        # 计算与干扰嵌入的相似度
        # gaze_clip: (B, 512)
        # distractor_embeddings: (D, 512)
        similarity = torch.matmul(gaze_clip, self.distractor_embeddings.T)  # (B, D)
        
        # 推离干扰方向
        # 如果相似度高，向相反方向调整
        distractor_direction = torch.matmul(
            similarity, 
            self.distractor_embeddings
        ) / self.distractor_embeddings.size(0)  # (B, 512)
        
        # 减去干扰成分
        purified = gaze_clip - 0.3 * distractor_direction
        
        # 正交化（确保与干扰方向正交）
        purified = self.disentangle(purified)
        
        return purified


class LISA(nn.Module):
    """LISA完整模型
    
    论文：Language-guided Interference-aware Spatial-Frequency Attention
    """
    
    def __init__(self, config=None):
        super().__init__()
        
        self.config = config or {}
        
        # 空域骨干
        import torchvision.models as models
        self.backbone = models.resnet18(pretrained=True)
        self.backbone = nn.Sequential(*list(self.backbone.children())[:-2])
        
        # FAM融合
        self.fam = FrequencyAttentionModulation(spatial_channels=512)
        
        # 语义解耦
        self.sdm = SemanticDisentanglement()
        
        # 视线预测头
        self.gaze_head = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),
            nn.Flatten(),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Linear(128, 2)  # 水平/垂直角度
        )
    
    def forward(self, img):
        """
        Args:
            img: (B, 3, 224, 224) 驾驶员图像
        
        Returns:
            output: dict
                - gaze: (B, 2) 视线方向
                - purified_feat: (B, 512) 纯净化特征
        """
        # 1. 空域特征提取
        spatial_feat = self.backbone(img)  # (B, 512, 7, 7)
        
        # 2. FAM融合
        fused_feat = self.fam(spatial_feat, img)  # (B, 256, 7, 7)
        
        # 3. 视线特征提取
        gaze_feat = F.adaptive_avg_pool2d(fused_feat, 1).squeeze(-1).squeeze(-1)  # (B, 256)
        
        # 4. 语义解耦
        purified_feat = self.sdm(gaze_feat[:, :128])  # (B, 512)
        
        # 5. 视线预测
        gaze = self.gaze_head(fused_feat)  # (B, 2)
        
        return {
            'gaze': gaze,
            'purified_feat': purified_feat,
            'fused_feat': fused_feat
        }


# 测试模型
if __name__ == "__main__":
    model = LISA()
    
    # 模拟输入
    img = torch.randn(4, 3, 224, 224)
    
    # 前向传播
    output = model(img)
    
    print(f"视线预测: {output['gaze']}")
    print(f"纯净化特征形状: {output['purified_feat'].shape}")

---

实验结果

性能对比

模型	正常条件误差	遮挡条件误差	光照变化误差
ResNet-18	4.2°	8.5°	6.8°
Transformer	3.8°	7.2°	5.5°
LISA	3.5°	5.5°	4.2°

遮挡场景细分

遮挡类型	传统方法	LISA	提升
墨镜	9.2°	5.8°	-37%
口罩	7.8°	5.2°	-33%
低光照	6.5°	4.3°	-34%
强光	7.1°	4.5°	-37%

---

消融实验

组件	误差	说明
仅空域	4.8°	基线
+频域融合	4.2°	-12.5%
+空间门控	3.9°	-7.1%
+语义解耦	3.5°	-10.3%

---

IMS开发启示

1. 系统集成方案

graph LR
    A[DMS摄像头] --> B[LISA模型]
    
    B --> C{干扰检测}
    
    C --> D[墨镜检测]
    C --> E[口罩检测]
    C --> F[光照异常]
    
    D --> G[启用频域分支]
    E --> G
    F --> G
    
    G --> H[鲁棒视线估计]

2. 硬件配置

组件	需求
摄像头	RGB-IR双模
处理器	QCS8255（频域FFT需GPU）
内存	≥4GB（CLIP模型加载）

3. 与Euro NCAP对接

ENCAP场景	LISA优势
强光驾驶	✅ 频域稳定
夜间驾驶	✅ 低光照鲁棒
墨镜用户	✅ 语义解耦
口罩用户	✅ 特征净化

4. 部署优化

# 部署优化：频域计算可预计算

class OptimizedLISA(LISA):
    def __init__(self):
        super().__init__()
        
        # 预计算干扰嵌入
        self.distractor_cache = None
    
    def forward_fast(self, img, precomputed_freq=None):
        """快速推理模式"""
        
        # 1. 使用预计算的频域特征（如果有）
        if precomputed_freq is not None:
            fused_feat = self.fam(self.backbone(img), precomputed_freq)
        else:
            fused_feat = self.fam(self.backbone(img), img)
        
        # 2. 简化解耦（缓存干扰嵌入）
        gaze = self.gaze_head(fused_feat)
        
        return gaze

---

参考文献

1. arXiv 2605.17287, “LISA: Language-guided Interference-aware Spatial-Frequency Attention”, 2026
2. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision”, ICML 2021

---

本文为LISA论文的详细解读与代码实现，面向IMS开发者提供墨镜/口罩遮挡和极端光照下的鲁棒视线估计方案。