來源 |  知乎專欄(黃浴）

關(guān)于傳感器融合，特別是攝像頭、激光雷達和雷達的前融合和和特征融合，是一個引人注意的方向。

1 “YOdar: Uncertainty-based Sensor Fusion for Vehicle Detection with Camera and Radar Sensors“， 11，2020

基于不確定性的融合方法。后處理采用gradient boosting，視覺來自YOLOv3，雷達來自1D segmentation network。

FCN-8 inspired radar network

Image of a radar detection example with four predicted slice bundles

YOdar

2 “Warping of Radar Data into Camera Image for Cross-Modal Supervision in Automotive Applications”，12，2020

將雷達的range-Doppler (RD) spectrum投射到攝像頭平面。由于設(shè)計的warping函數(shù)可微分，所以在訓(xùn)練框架下做BP。該warping操作依賴于環(huán)境精確的scene flow，故提出一個來自激光雷達、攝像頭和雷達的scene flow估計方法，以提高warping操作精度。實驗應(yīng)用涉及了direction-of-arrival (DoA) estimation, target detection, semantic segmentation 和 estimation of radar power from camera data等。

model pipeline

DRISFwR overview （deep rigid instance scene flow with radar）

Automatic scene flow alignment to Radar data via DRISFwR：

RGB image and RD-map with two vehicles

Scale-space of radar data used in DRISFwR with energy & partial derivative

Power projections

RD-map warping into camera image：

Loss in scale-space：

最后實驗結(jié)果比較：

Qualitative results of target detection on test data examples

Qualitative results of semantic segmentation on test data examples

Overview of the model pipeline for camera based estimators for NN training：

Qualitative results of SNR prediction on test data：

3 "RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localization", 2 2021

雷達目標(biāo)檢測網(wǎng)絡(luò)RODNet，但訓(xùn)練是通過一個攝像頭-雷達監(jiān)督算法，無需標(biāo)注，可實現(xiàn)射頻（RF）圖像的實時目標(biāo)檢測。原始毫米波雷達信號轉(zhuǎn)換為range-azimuth坐標(biāo)的RF圖像；RODNet預(yù)測雷達FoV的目標(biāo)似然性。兩個定制的模塊M-Net和temporal deformable convolution分別處理multi-chirp merging信息以及目標(biāo)相對運動。訓(xùn)練中采用camera-radar fusion (CRF) 策略，另外還建立一個新數(shù)據(jù)集CRUW1。

cross-modal supervision pipeline for radar object detection in a teacher-student platform

workflow of the RF image generation from the raw radar signals

The architecture and modules of RODNet

Three teacher’s pipelines for cross-model supervision

temporal inception convolution layer

4 “Radar Camera Fusion via Representation Learning in Autonomous Driving”，4，2021

重點討論data association問題。而rule-based association methods問題較多，故此討論radar-camera association via deep representation learning 以開發(fā)特征級的交互和全局推理。將檢測結(jié)果轉(zhuǎn)換成圖像通道，和原圖像一起送入一個深度CNN模型，即AssociationNet。另外，設(shè)計了一個loss sampling mechanism 和 ordinal loss 來克服不完美的標(biāo)注困難，確保一個類似人工的推理邏輯。

associations between radardetections (radar pins) and camera detections (2D bounding boxes).

AssociationNet

architecture of the neural network

process of obtaining final associationsfrom the learned representation vectors

illustration of radar pins, bounding boxes, and association relationships under BEV perspective

the red solid lines represent the true-positive associations; and the pink solid lines represent predicted positive associations but labeled as uncertain in the ground-truth

The added green lines represent the false-positive predictions; and the added black lines represent the false-negative predictions