BEV感知Occupancy网络3D目标检测与占据栅格预测一、引言自动驾驶的核心挑战之一是精确理解周围的三维环境。传统2D检测丢失深度信息、LiDAR昂贵且稀疏。BEVBird’s Eye View感知将多视角相机图像统一变换到鸟瞰视角实现精确的3D检测。Occupancy网络更进一步预测每个体素是否被占据处理任意形状障碍物。本文将深入 BEVFormer 架构、从相机到BEV的几何变换、以及 Occ3D 占据栅格预测。二、相机→BEV 几何变换2.1 IPM逆透视变换importnumpyasnpimportcv2defcompute_ipm_matrix(K,R,T,H3,W50,resolution0.1):计算逆透视变换矩阵 K: 内参 [3,3], R: 旋转 [3,3], T: 平移 [3] 输出: BEV尺寸 H×W分辨率 resolution 米/像素# 构建外参矩阵RTnp.hstack([R,T.reshape(3,1)])# 定义地面平面点在世界坐标系 z0 平面上grid_x(np.arange(W)-W//2)*resolution grid_y(np.arange(H)-H//2)*resolution grid_xx,grid_yynp.meshgrid(grid_x,grid_y)# 世界坐标 → 相机坐标world_pointsnp.stack([grid_xx.flatten(),grid_yy.flatten(),np.zeros_like(grid_xx.flatten()),np.ones_like(grid_xx.flatten())],axis0)cam_pointsRT world_points# [3, N]# 相机坐标 → 像素坐标pixel_pointsK cam_points pixel_pointspixel_points[:2]/pixel_points[2]# 透视除法returnpixel_points.reshape(2,H,W).transpose(1,2,0)2.2 Lift-Splat-ShootBEVDet核心classLiftSplatShoot(nn.Module):LSS: 从多视角图像提升到3D再投影到BEVdef__init__(self,frustum_depth60,grid_size(200,200)):super().__init__()self.Dfrustum_depth# 深度离散化1-60米self.H,self.Wgrid_size# 深度分布网络每个像素预测深度概率self.depth_netnn.Sequential(nn.Conv2d(256,256,3,padding1),nn.ReLU(),nn.Conv2d(256,self.D,1)# 输出D类深度分布)# 特征网络self.feature_netnn.Conv2d(256,64,1)defforward(self,images,extrinsics,intrinsics):images: [B,N,3,H,W] N个相机B,N,C,H,Wimages.shape# 1. Lift: 每个像素 → 3D视锥featuresself.encode(images)# [B,N,C,H,W]# 深度分布depth_probsself.depth_net(features).softmax(dim1)# [B*N,D,H,W]# 特征与深度外积pixel_featuresself.feature_net(features)# [B*N,64,H,W]# 提升到3D: [B*N,D,H,W] × [B*N,64,H,W] → [B*N,64,D,H,W]frustum_featuresdepth_probs.unsqueeze(1)*pixel_features.unsqueeze(2)# 2. Splat: 3D视锥 → BEV (相机坐标 → 世界BEV坐标)bev_gridself.compute_bev_grid(extrinsics,intrinsics,B,N)# 使用视锥点在BEV坐标系的位置将特征 scatter 到BEV网格bev_featuresself.splat_to_bev(frustum_features,bev_grid)returnbev_featuresdefsplat_to_bev(self,frustum_features,bev_indices):将3D视锥 splat 到2D BEV平面求和池化B,C,D,H,Wfrustum_features.shape bevtorch.zeros(B,C,self.H,self.W,devicefrustum_features.device)# 使用scatter_add高效实现forbinrange(B):indicesbev_indices[b]# [D,H,W,2] → (x_idx, y_idx)fordinrange(D):# 过滤有效坐标valid(indices[d,:,:,0]0)(indices[d,:,:,0]self.W)\(indices[d,:,:,1]0)(indices[d,:,:,1]self.H)# Sum poolingidxindices[d,valid]bev[b].index_put_((idx[:,1],idx[:,0]),frustum_features[b,:,d][:,valid],accumulateTrue)returnbev三、BEVFormer 架构classBEVFormer(nn.Module):BEVFormer: 使用时空Transformer的BEV感知def__init__(self,bev_h200,bev_w200,embed_dim256):super().__init__()self.bev_h,self.bev_wbev_h,bev_w# BEV查询可学习的位置编码self.bev_queriesnn.Parameter(torch.randn(bev_h*bev_w,embed_dim))# 空间交叉注意力Image → BEVself.spatial_cross_attentionSpatialCrossAttention(embed_dim)# 时间自注意力融合历史BEV特征self.temporal_self_attentionTemporalSelfAttention(embed_dim)# BEV编码器层self.encoder_layersnn.ModuleList([BEVFormerLayer(embed_dim)for_inrange(6)])defforward(self,multi_view_features,prev_bevNone):multi_view_features: [B,N,C,H,W]Bmulti_view_features.shape[0]# 1. 初始化BEV查询bev_queriesself.bev_queries.unsqueeze(0).repeat(B,1,1)# 2. 时间自注意力融合历史BEVifprev_bevisnotNone:bev_queriesself.temporal_self_attention(bev_queries,prev_bev)# 3. 空间交叉注意力 FFN多层堆叠forlayerinself.encoder_layers:bev_querieslayer(bev_queries,multi_view_features)# 4. 重塑为BEV特征图bev_featuresbev_queries.reshape(B,self.bev_h,self.bev_w,-1)returnbev_featuresclassSpatialCrossAttention(nn.Module):空间交叉注意力: BEV查询 → 参考点 → 采样图像特征def__init__(self,embed_dim256,num_points4):super().__init__()self.embed_dimembed_dim self.num_pointsnum_points# 可变形注意力self.deformable_attentionDeformableAttention(embed_dim)# 采样偏移预测self.sampling_offsetsnn.Linear(embed_dim,num_points*2)self.attention_weightsnn.Linear(embed_dim,num_points)defforward(self,bev_queries,image_features,reference_points): bev_queries: [B, N_bev, C] reference_points: [B, N_bev, num_views, 2]每个BEV点在每个相机视角的投影坐标 B,N_bev,Cbev_queries.shape N_viewimage_features.shape[1]# 预测采样偏移和注意力权重offsetsself.sampling_offsets(bev_queries)# [B,N_bev,4*2]attn_weightsself.attention_weights(bev_queries).softmax(-1)# 在图像特征上采样sampled_features[]forvinrange(N_view):# 获取该视角的参考点ref_ptsreference_points[:,:,v]# [B,N_bev,2]# 2D grid_sampleptsref_pts.unsqueeze(2)offsets.reshape(B,N_bev,self.num_points,2)ptspts*2-1# 归一化到[-1,1]featF.grid_sample(image_features[:,v],# [B,C,H,W]pts,# [B,N_bev,4,2]align_cornersTrue,modebilinear)# [B,C,N_bev,4]sampled_features.append(feat)# 加权融合sampledtorch.stack(sampled_features,dim-1)# [B,C,N_bev,4,N_view]weighted(sampled*attn_weights.unsqueeze(1).unsqueeze(-1)).sum(-1).sum(-1)returnweighted.transpose(1,2)# [B,N_bev,C]四、Occupancy 占据栅格4.1 Occ3D 数据# Occupancy标注格式: 每个体素一个标签# 标签集: {0:free, 1:car, 2:truck, 3:pedestrian, ..., 16:traffic_cone}classOcc3DDecoder(nn.Module):从BEV特征解码占据栅格def__init__(self,bev_dim256,occ_size(200,200,16),num_classes17):super().__init__()self.occ_h,self.occ_w,self.occ_zocc_size# 高度上采样BEV → 3Dself.height_upnn.Sequential(nn.ConvTranspose3d(bev_dim,128,(4,1,1),stride(2,1,1)),nn.BatchNorm3d(128),nn.ReLU(),nn.ConvTranspose3d(128,64,(4,1,1),stride(2,1,1)),)# 分割头self.seg_headnn.Sequential(nn.Conv3d(64,64,3,padding1),nn.BatchNorm3d(64),nn.ReLU(),nn.Conv3d(64,num_classes,1))defforward(self,bev_features):bev_features: [B,C,H,W]# BEV → 3D: 沿Z轴堆叠B,C,H,Wbev_features.shape occbev_features.unsqueeze(2).repeat(1,1,self.occ_z,1,1)# 3D卷积上采样occself.height_up(occ)# [B,64,64,H,W]# 占据预测occ_logitsself.seg_head(occ)# [B,17,64,H,W]returnocc_logits# 损失函数ce_lossnn.CrossEntropyLoss(ignore_index255)# 忽略未知lovasz_lossLovaszSoftmax()# 处理类别不平衡lossce_loss(pred,gt)0.5*lovasz_loss(pred,gt)五、端到端3D检测器classBEV3DDetector(nn.Module):基于BEV的端到端3D目标检测器def__init__(self):self.backboneResNet50()self.neckFPN([256,512,1024,2048],256)self.view_transformLiftSplatShoot()self.bev_encoderBEVFormer()# 检测头类似CenterPointself.heatmap_headnn.Sequential(nn.Conv2d(256,256,3,padding1),nn.BatchNorm2d(256),nn.ReLU(),nn.Conv2d(256,10,1)# 10类)self.reg_headnn.Conv2d(256,8,1)# x,y,z,dx,dy,dz,sinθ,cosθdefforward(self,images,calibs):# 图像特征提取img_featsself.backbone(images)img_featsself.neck(img_feats)# 视角变换bev_featsself.view_transform(img_feats,calibs)bev_featsself.bev_encoder(bev_feats)# 检测heatmapself.heatmap_head(bev_feats)regself.reg_head(bev_feats)returnheatmap,reg# NMS后处理defdecode_boxes(heatmap,reg,threshold0.3):解码BEV检测结果# 1. 热力图峰值检测NMS在BEV空间peaksnms_2d(heatmap,kernel3)# 2. 提取检测框boxes[]forpeakinpeaks:cx,cypeak# 回归参数dx,dy,z,length,width,height,sin_y,cos_yreg[:,cx,cy]rot_ytorch.atan2(sin_y,cos_y)boxes.append({center:(cxdx,cydy,z),dims:(length,width,height),yaw:rot_y,score:heatmap[:,cx,cy].max()})returnboxes六、数据增强与训练技巧# BEV专用数据增强classBEVAugmentation:staticmethoddefrandom_flip(images,boxes,calibs,p0.5):左右翻转ifrandom.random()p:imagestorch.flip(images,dims[-1])boxes[:,0]-boxes[:,0]boxes[:,6]-boxes[:,6]# yawreturnimages,boxes,calibsstaticmethoddefrandom_rotate(bev_feats,boxes,angle_range(-22.5,22.5)):BEV空间随机旋转anglerandom.uniform(*angle_range)# 旋转BEV特征图Mcv2.getRotationMatrix2D((W/2,H/2),angle,1.0)bev_featscv2.warpAffine(bev_feats,M,(W,H))# 旋转检测框boxes[:,6]np.radians(angle)returnbev_feats,boxes七、主流框架对比框架输入输出速度NDS(mAP)BEVDet多相机3D框25fps0.488BEVDepth相机深度3D框20fps0.535BEVFormer多相机时序3D框5fps0.569BEVFormer v2相机LiDAR3D框3fps0.614Occ3D多相机时序占据栅格3fps0.485(mIoU)八、总结BEV感知核心要点视角变换是关键— LSS的Lift-Splat-Shoot是工程核心Transformer是标配— 空间交叉注意力 时间自注意力Occupancy是未来— 从3D框到稠密占据栅格处理任意形状障碍物时序融合提升大— 单帧→多帧mAP提升5-10个点