LISA：语言引导的频域融合视线估计方法

论文： LISA: Language-guided Interference-aware Spatial-Frequency Attention for Driver Gaze Estimation
作者： Zhenye Yang, Ruichen Zhou, Pei Zhang, Huan Li, Jinpeng Chen
机构： 北京邮电大学、广西大学、浙江大学
发表： arXiv 2026
链接： https://arxiv.org/abs/2605.17287

---

核心创新

LISA 首次提出频域先验 + 语言引导解耦的视线估计框架，解决两大核心挑战：

挑战	传统方法问题	LISA 解决方案
空间错位	光照变化导致像素特征不稳定	频域先验（幅度谱稳定）
外观耦合	墨镜/口罩等属性与视线特征混淆	语言引导解耦（CLIP + 正交约束）

一句话总结： 利用频域稳定性抵抗光照变化，用CLIP语言模型分离视线特征与外观干扰。

---

方法详解

1. 整体架构

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

class LISA(nn.Module):
    """
    LISA: Language-guided Interference-aware Spatial-Frequency Attention
    
    核心组件：
    1. FAM Fusion - 频域注意力融合
    2. SDM - 语义解耦模块（CLIP引导）
    """
    
    def __init__(self, config: dict):
        super().__init__()
        
        # 骨干网络（ResNet-18）
        self.backbone = ResNet18Backbone()
        
        # FAM融合模块
        self.fam_fusion = FAMFusion(
            spatial_dim=512,
            frequency_dim=512
        )
        
        # 空间显著性门控
        self.spatial_gate = SpatialSaliencyGating(512)
        
        # 语义解耦模块
        self.sdm = SemanticDisentanglementModule(
            feature_dim=512,
            clip_model='ViT-B/32'
        )
        
        # 视线回归头
        self.gaze_regressor = nn.Sequential(
            nn.Linear(512, 256),
            nn.ReLU(inplace=True),
            nn.Dropout(0.3),
            nn.Linear(256, 2)  # pitch, yaw
        )
        
    def forward(self, image: torch.Tensor) -> dict:
        """
        Args:
            image: (B, 3, H, W) RGB图像
            
        Returns:
            gaze: (B, 2) 视线角度 [pitch, yaw]
            features: 解耦后的特征
        """
        # 1. 骨干特征提取
        spatial_feat = self.backbone(image)
        
        # 2. FAM融合（空间+频域）
        fused_feat = self.fam_fusion(spatial_feat, image)
        
        # 3. 空间显著性门控
        gated_feat = self.spatial_gate(fused_feat)
        
        # 4. 语义解耦
        disentangled_feat, disentangle_loss = self.sdm(gated_feat)
        
        # 5. 视线回归
        gaze = self.gaze_regressor(disentangled_feat)
        
        return {
            'gaze': gaze,
            'features': disentangled_feat,
            'disentangle_loss': disentangle_loss
        }


class FAMFusion(nn.Module):
    """
    Frequency-Attention Modulated Fusion
    
    将频域稳定特征注入空间特征
    """
    
    def __init__(self, spatial_dim: int, frequency_dim: int):
        super().__init__()
        
        # 频谱注入块
        self.spectral_injection = SpectralInjectionBlock(spatial_dim)
        
        # 融合层
        self.fusion = nn.Sequential(
            nn.Linear(spatial_dim + frequency_dim, spatial_dim),
            nn.ReLU(inplace=True)
        )
        
    def forward(self, spatial_feat: torch.Tensor, image: torch.Tensor) -> torch.Tensor:
        """
        Args:
            spatial_feat: (B, C, H, W) 空间特征
            image: (B, 3, H, W) 原始图像
            
        Returns:
            fused: (B, C, H, W) 融合特征
        """
        # FFT转换
        freq_feat = self._extract_frequency_features(image)
        
        # 频谱注入
        injected_feat = self.spectral_injection(spatial_feat, freq_feat)
        
        return injected_feat
    
    def _extract_frequency_features(self, image: torch.Tensor) -> torch.Tensor:
        """提取频域特征"""
        B, C, H, W = image.shape
        
        # FFT
        freq = torch.fft.fft2(image, dim=(-2, -1))
        amp = torch.abs(freq)  # 幅度谱
        phase = torch.angle(freq)  # 相位谱
        
        # 对数幅度（稳定化）
        log_amp = torch.log(amp + 1e-8)
        
        # 低频提取
        low_freq = self._extract_low_frequency(log_amp, ratio=0.1)
        
        return low_freq
    
    def _extract_low_frequency(self, amp: torch.Tensor, ratio: float) -> torch.Tensor:
        """提取低频分量"""
        B, C, H, W = amp.shape
        
        # 中心化
        amp_shift = torch.fft.fftshift(amp, dim=(-2, -1))
        
        # 掩码
        mask = torch.zeros_like(amp_shift)
        center_h, center_w = H // 2, W // 2
        radius_h, radius_w = int(H * ratio), int(W * ratio)
        
        mask[:, :, 
             center_h-radius_h:center_h+radius_h,
             center_w-radius_w:center_w+radius_w] = 1.0
        
        # 应用掩码
        low_freq = amp_shift * mask
        
        return low_freq


class SpectralInjectionBlock(nn.Module):
    """频谱注入块"""
    
    def __init__(self, channels: int):
        super().__init__()
        
        self.channel_attention = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),
            nn.Conv2d(channels, channels // 16, 1),
            nn.ReLU(inplace=True),
            nn.Conv2d(channels // 16, channels, 1),
            nn.Sigmoid()
        )
        
    def forward(self, spatial_feat: torch.Tensor, freq_feat: torch.Tensor) -> torch.Tensor:
        """
        将频域特征注入空间特征
        """
        # 频域特征上采样
        freq_up = F.interpolate(freq_feat, size=spatial_feat.shape[-2:], 
                                  mode='bilinear', align_corners=False)
        
        # 通道注意力
        attention = self.channel_attention(spatial_feat)
        
        # 注入
        injected = spatial_feat + attention * freq_up
        
        return injected


class SpatialSaliencyGating(nn.Module):
    """空间显著性门控"""
    
    def __init__(self, channels: int):
        super().__init__()
        
        self.gate = nn.Sequential(
            nn.Conv2d(channels, 1, 1),
            nn.Sigmoid()
        )
        
    def forward(self, feat: torch.Tensor) -> torch.Tensor:
        """
        突出眼部区域
        """
        saliency_map = self.gate(feat)
        return feat * saliency_map


class SemanticDisentanglementModule(nn.Module):
    """
    Semantic Disentanglement Module
    
    使用CLIP分离视线特征与外观干扰
    """
    
    # 干扰文本模板
    DISTRACTOR_TEMPLATES = [
        "a driver wearing sunglasses",
        "a driver wearing a mask",
        "a driver with glasses",
        "a driver with hat"
    ]
    
    def __init__(self, feature_dim: int, clip_model: str = 'ViT-B/32'):
        super().__init__()
        
        # CLIP编码器（冻结）
        self.clip_encoder = self._load_clip(clip_model)
        
        # 文本嵌入（预计算）
        self.text_embeddings = self._encode_distractors()
        
        # 特征投影
        self.projector = nn.Linear(feature_dim, 512)
        
    def forward(self, features: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Args:
            features: (B, D) 输入特征
            
        Returns:
            disentangled: 解耦特征
            loss: 解耦损失
        """
        # 投影到CLIP空间
        projected = self.projector(features)
        
        # 计算与干扰文本的相似度
        similarity = F.cosine_similarity(
            projected.unsqueeze(1),
            self.text_embeddings.unsqueeze(0),
            dim=-1
        )  # (B, num_distractors)
        
        # Push-Away损失：最小化与干扰的相似度
        push_loss = similarity.mean()
        
        # 正交约束：视线特征应与外观特征正交
        ortho_loss = self._orthogonal_loss(projected)
        
        total_loss = push_loss + 0.1 * ortho_loss
        
        return features, total_loss
    
    def _load_clip(self, model_name: str):
        """加载CLIP模型"""
        import clip
        model, _ = clip.load(model_name, device='cuda')
        
        # 冻结参数
        for param in model.parameters():
            param.requires_grad = False
        
        return model
    
    def _encode_distractors(self) -> torch.Tensor:
        """编码干扰文本"""
        import clip
        
        text_tokens = clip.tokenize(self.DISTRACTOR_TEMPLATES).cuda()
        with torch.no_grad():
            text_features = self.clip_encoder.encode_text(text_tokens)
        
        return text_features.float()
    
    def _orthogonal_loss(self, features: torch.Tensor) -> torch.Tensor:
        """正交约束损失"""
        # 特征应与干扰嵌入正交
        ortho = torch.mm(features, self.text_embeddings.T)
        loss = (ortho ** 2).mean()
        
        return loss


# 测试代码
if __name__ == "__main__":
    model = LISA({})
    model.eval()
    
    # 模拟输入
    image = torch.randn(2, 3, 224, 224)
    
    with torch.no_grad():
        result = model(image)
    
    print(f"视线预测: {result['gaze']}")
    print(f"解耦损失: {result['disentangle_loss']:.4f}")

---

实验结果

基准测试

方法	ETH-XGaze	DADA-2000	平均
GazeCLR	11.2°	15.8°	13.5°
PureGaze	10.8°	15.2°	13.0°
LISA	9.1°	13.5°	11.3°

鲁棒性测试

干扰类型	传统方法	LISA	提升
正常光照	6.5°	5.8°	10.8%
弱光	14.2°	8.9°	37.3%
强光	12.8°	9.2°	28.1%
墨镜	18.5°	12.4°	33.0%
口罩	16.2°	11.8°	27.2%

---

IMS应用启示

1. 夜间驾驶场景

LISA的频域先验特别适合夜间光照变化：

class LISA_NightAdapter:
    """LISA夜间适配"""
    
    def __init__(self, model_path: str):
        self.model = LISA({})
        self.model.load_state_dict(torch.load(model_path))
        
    def estimate_gaze(self, image: np.ndarray, is_night: bool = False) -> dict:
        """
        夜间场景视线估计
        
        LISA频域融合自动处理光照变化
        """
        # 预处理
        x = self._preprocess(image)
        
        # 推理
        with torch.no_grad():
            result = self.model(x)
        
        return {
            'gaze_pitch': result['gaze'][0, 0].item(),
            'gaze_yaw': result['gaze'][0, 1].item(),
            'confidence': 1.0 - result['disentangle_loss'].item()
        }

2. 墨镜驾驶员检测

语义解耦模块专门处理墨镜干扰：

class SunglassesRobustDetector:
    """墨镜鲁棒检测"""
    
    def __init__(self):
        self.model = LISA({})
        # SDM模块已学习"墨镜"干扰特征
        
    def detect_with_sunglasses(self, image: np.ndarray) -> dict:
        """
        墨镜场景视线估计
        
        SDM模块自动分离"墨镜"属性
        """
        result = self.model(image)
        
        # SDM确保视线特征与墨镜外观正交
        return result

---

技术路线图

短期（1-3个月）

任务	输入	输出	验证
模型训练	ETH-XGaze	LISA-DMS.pth	误差 < 10°
ONNX导出	训练模型	lisa.onnx	延迟 < 25ms
墨镜测试	戴墨镜数据	准确率报告	误差 < 13°

中期（3-6个月）

任务	目标	验证
夜间测试	误差 < 10°	实车夜间测试
芯片部署	延迟 < 30ms	QCS8255测试
Euro NCAP验证	通过所有场景	第三方测试

---

参考资料

1. Yang et al., “LISA: Language-guided Interference-aware Spatial-Frequency Attention for Driver Gaze Estimation”, arXiv 2026
2. CLIP: Radford et al., “Learning Transferable Visual Models From Natural Language Supervision”, ICML 2021
3. ETH-XGaze Dataset

---

关键词： 视线估计, 频域融合, 语言引导, LISA, 墨镜鲁棒, Euro NCAP

发布时间： 2026-07-22

作者： OpenClaw AI Research