GazeCapsNet轻量级视线估计：20ms实时推理的边缘部署方案

论文来源： Sensors 2025 (PMC11860563)
核心创新： Capsule Network轻量级设计
性能指标： 20ms推理延迟，11.7M参数

---

论文核心贡献

1. 胶囊网络架构

传统CNN丢失空间关系，Capsule Network保留姿态信息，更适合视线估计。

2. 轻量化设计

指标	GazeCapsNet	传统CNN
参数量	11.7M	50M+
推理时间	20ms	50ms+
精度	4.2°	3.8°

---

实现代码

"""
GazeCapsNet轻量级视线估计
"""
import torch
import torch.nn as nn
import numpy as np

class CapsuleLayer(nn.Module):
    """胶囊层"""
    
    def __init__(self, in_capsules: int, out_capsules: int, 
                 in_dim: int, out_dim: int):
        super().__init__()
        self.in_capsules = in_capsules
        self.out_capsules = out_capsules
        self.in_dim = in_dim
        self.out_dim = out_dim
        
        # 权重矩阵
        self.weight = nn.Parameter(
            torch.randn(in_capsules, out_capsules, in_dim, out_dim)
        )
        
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        前向传播
        
        Args:
            x: 输入胶囊 (B, in_capsules, in_dim)
        
        Returns:
            out: 输出胶囊 (B, out_capsules, out_dim)
        """
        # 计算预测向量
        predictions = torch.einsum('bci,ciop->bcop', x, self.weight)
        
        # 动态路由
        out = self._dynamic_routing(predictions)
        
        return out
    
    def _dynamic_routing(self, predictions: torch.Tensor, 
                         iterations: int = 3) -> torch.Tensor:
        """动态路由算法"""
        batch_size = predictions.shape[0]
        
        # 初始化耦合系数
        coupling = torch.zeros(batch_size, self.in_capsules, 
                              self.out_capsules)
        
        for _ in range(iterations):
            # 计算注意力
            attn = torch.softmax(coupling, dim=-1)
            
            # 加权求和
            s = torch.sum(attn.unsqueeze(-1) * predictions, dim=1)
            
            # Squash激活
            v = self._squash(s)
            
            # 更新耦合系数
            coupling = coupling + torch.sum(predictions * v.unsqueeze(1), dim=-1)
        
        return v
    
    def _squash(self, x: torch.Tensor) -> torch.Tensor:
        """Squash激活函数"""
        norm = torch.norm(x, dim=-1, keepdim=True)
        return (norm ** 2 / (1 + norm ** 2)) * x / (norm + 1e-8)


class GazeCapsNet(nn.Module):
    """GazeCapsNet视线估计网络"""
    
    def __init__(self):
        super().__init__()
        
        # 特征提取
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(64, 128, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, 256, 3, stride=2, padding=1),
            nn.ReLU()
        )
        
        # 主胶囊层
        self.primary_caps = CapsuleLayer(
            in_capsules=256, out_capsules=32, 
            in_dim=1, out_dim=8
        )
        
        # 数字胶囊层（输出视线）
        self.gaze_caps = CapsuleLayer(
            in_capsules=32, out_capsules=1,
            in_dim=8, out_dim=3  # (pitch, yaw, confidence)
        )
        
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        前向传播
        
        Args:
            x: 输入图像 (B, 3, H, W)
        
        Returns:
            gaze: 视线向量 (B, 3)
        """
        # 特征提取
        features = self.features(x)
        
        # 展平为胶囊
        batch_size = features.shape[0]
        capsules = features.view(batch_size, 256, -1).mean(dim=-1)
        capsules = capsules.unsqueeze(-1)
        
        # 胶囊网络
        primary = self.primary_caps(capsules.squeeze(-1))
        gaze = self.gaze_caps(primary)
        
        return gaze.squeeze(1)
    
    def predict_gaze(self, image: np.ndarray) -> dict:
        """预测视线"""
        # 预处理
        x = self._preprocess(image)
        
        # 推理
        with torch.no_grad():
            gaze = self.forward(x)
        
        # 提取pitch/yaw
        pitch = gaze[0, 0].item()
        yaw = gaze[0, 1].item()
        confidence = gaze[0, 2].item()
        
        return {
            'pitch': pitch,
            'yaw': yaw,
            'confidence': confidence
        }
    
    def _preprocess(self, image: np.ndarray) -> torch.Tensor:
        """图像预处理"""
        # 缩放到224x224
        from PIL import Image
        img = Image.fromarray(image).resize((224, 224))
        x = np.array(img).astype(np.float32) / 255.0
        x = torch.from_numpy(x).permute(2, 0, 1).unsqueeze(0)
        return x


# 测试
if __name__ == "__main__":
    model = GazeCapsNet()
    model.eval()
    
    # 模拟输入
    dummy = np.random.randint(0, 255, (480, 640, 3), dtype=np.uint8)
    result = model.predict_gaze(dummy)
    
    print(f"视线估计: pitch={result['pitch']:.1f}°, yaw={result['yaw']:.1f}°")
    print(f"置信度: {result['confidence']:.2f}")

---

边缘部署优化

INT8量化

指标	FP32	INT8
模型大小	46MB	12MB
推理时间	20ms	8ms
精度损失	-	<1%

---

IMS开发启示

优先级	功能	原因
P0	胶囊网络实现	核心架构
P0	INT8量化	边缘部署
P1	多任务扩展	分心检测

---

参考论文：

• Sensors 2025: “GazeCapsNet: A Lightweight Gaze Estimation Framework”