Automatic Generation of Labelled mmWave FMCW Radar Datasets Using Collocated Optical Sensors and Computer Vision

Bourn, William

Automatic Generation of Labelled mmWave FMCW Radar Datasets Using Collocated Optical Sensors and Computer Vision

Thesis / Dissertation

2026

Publisher

University of Cape Town

Department

Department of Electrical Engineering

Faculty

Faculty of Engineering and the Built Environment

Abstract

Advancements made in computational power, the development of complex AI algorithms, and an industry push to create self-driving cars and autonomous systems has resulted in a large amount of research into computer vision. The majority of computer vision research is performed with optical sensors because radar and lidar sensors suffer from a paucity of data that makes classification tasks difficult, and lack easily implemented open-source datasets for general applications. Future research into radar-based computer vision would benefit from a method of automating the process of annotating data, which can be implemented using sensor fusion and existing image recognition algorithms. We propose a processing chain that uses fused spatio-optical and radar data to automatically produce class and localisation ground truth labels in radar data using a YOLO image recognition model to label objects in a projection of 3D optical space. We developed a processing pipeline to perform automatic labelling of radar data and a prototype data capture rig. We analysed the performance of the pipeline by processing data captured by the rig into annotated radar range-azimuth-Doppler frames for human classification and training a CNN classification model on the data. Furthermore, we analysed the performance of the approach to calibration in the processing chain. The CNN classifier achieved strong human classification performance, though was affected by localisation errors that were produced by the processing pipeline. Despite this, the strong classification performance indicates that the pipeline was able to successfully perform automatic annotation of radar range-azimuth-Doppler data with human target information. The processing chain has potential use in radar-based computer vision going forward, and the automatic labelling approach detailed in this report merits further development.

Keywords

computer vision

radar datasets

Reference:

Collections

Masters

Full item page