Scene Understanding in Deep Learning-Based End-to-End Controllers for Autonomous Vehicles

作者单位：Department of Vehicle Engineering, Jilin University；Department of Mechanical Engineering, Universityof California；State Key Laboratory of AutomotiveSimulation and Control, Jilin University

发表期刊：IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS: SYSTEMS, VOL. 49, NO. 1, JANUARY 2019

摘要：Deep learning techniques have been widely used inautonomous driving community for the purpose of environmentperception. Recently, it starts being adopted for learningend-to-end controllers for complex driving scenarios. However,the complexity and nonlinearity of the network architecture limits its interpretability to understand driving scenarios and judgethe importance of certain visual regions in sensory scenes. In thispaper, based on the convolutional neural network (CNN), we propose two complementary frameworks to automatically determinethe most contributive regions of the input scenes, offering intuitive knowledge of how a trained end-to-end autonomous vehiclecontroller understands driving scenarios. In the fifirst framework, a feature map-based method is proposed by leveragingcurrent progress in CNN visualization, in which the deconvolution approach recovers the feature maps to extract featuresthat contribute most to understand driving scenes. In the secondframework, the importance level of regions is ranked using theerror map between the labeled and predicted control inputs generated by occluding different parts of input scenes, thus providinga pixel-wise rank of importance. Test data sets with extractedcontributive regions are input to the CNN controller. Then, different CNN controllers trained with the new data sets preprocessedusing our proposed frameworks are verifified via closed-loop tests.Results show that both the features identifified from the first framework and the regions identifified from the second framework are of crucial importance to scene understanding for thecontroller and can signifificantly affect the performance of CNN controllers.

关键词：Autonomous vehicles, convolutional neuralnetwork (CNN), scene understanding.