Generating synthetic point clouds of real cities for semantic road space segmentation

About the TUM Chair of Photogrammetry and Remote Sensing:

The main focus of the chair is the development of methods for registration, segmentation, classification, and 3D reconstruction of images and 3D point clouds. Applications of such methods span from monitoring urban environments, reconstructing 3D city models, to providing testbeds for autonomous driving simulations, among others. In the case of this project, the Chair will provide support in point clouds segmentation for reconstructing 3D environments for testing autonomous driving.

About the TUM Chair of Geoinformatics:

The research field of the Chair focuses on developing methods for the spatial, temporal and semantic modelling, storage, analysis and visualisation of the environment. The Chair is involved in the development of the international standard CityGML, which is used for representing, storing and exchanging semantic 3D city models and issued by the Open Geospatial Consortium. Within the scope of this project, the Chair will provide semantic models as well as autonomous driving maps for selected streets of the city of Ingolstadt.

Project description

Deep neural networks are commonly used in robotics or autonomous driving to interpret sensor information. Since a wide variance of scenarios is critical for training neural networks, synthetically generated datasets are often utilised. However, this raises the question of whether synthetically generated training data positively influence the inference performance for real-world applications.

In order to investigate this question, georeferenced mobile laser scanning point clouds of selected streets in the city of Ingolstadt are provided for this project. Furthermore, reference maps for these streets are provided in the format ASAM OpenDRIVE, which vehicle manufacturers use to develop and test automated driving functions. These reference maps can be combined with semantic building models containing facade details to create semantic street space models, which then serve as a virtual environment for a driving simulation. In these environment models, a virtual drive with a sensor simulation will be conducted, thus obtaining a synthetic point cloud and its segmentation.

In order to evaluate the differences between the real-world point cloud and the synthetic point cloud, the goal of the project is to investigate and apply comparison metrics concerning their geometry as well as semantics. This will be possible since the real-world and the synthetic point clouds cover the same scene. The last project task intends to train state-of-the-art deep neural networks for point cloud segmentation: By varying the proportions of real-world and synthetically generated training data, the goal is to investigate the potential improvements in the network’s inference performance.

Objectives

1. Generating virtual environment for synthetic point clouds simulation:
   1. Goal: Creating a pipeline for generating synthetic, semantic 3D point clouds using provided semantic 3D road space models
   2. Tasks:
      1. Transforming high-definition semantic 3D road space models to simulation software (CARLA) while preserving the relevant semantics
      2. Running a virtual simulator of a 3D mobile laser scanner using high-definition semantic 3D road space models
   3. Key words: MLS point clouds, 3D sensor models, CARLA, OpenDRIVE, semantic 3D city and road space models, CityGML
2. Comparing synthetic point clouds vs. real point clouds:
   1. Goal: Obtaining evaluation metrics to efficiently compare geometric and semantic differences between synthetic and real point clouds
   2. Tasks:
      1. Annotating real-world point clouds with provided classes
      2. Finding appropriate metrics for point cloud’s comparison
      3. Comparing real and synthetic point clouds using selected metrics, e.g., point-to-point deviations (e.g., using Hausdorff distance) and semantic completeness (e.g., IoU)
   3. Key words: MLS point clouds, ground-truth datasets, understanding domain gaps, distance metrics, generating training data
3. 3D point cloud segmentation using deep neural networks:
   1. Goal: Segmenting 3D point clouds using the state-of-the-art deep learning networks and obtained point clouds
   2. Tasks:
      1. Selecting the state-of-the-art networks suitable for the task (e.g., Point Transformers)
      2. Training of networks on real and testing on synthetic point clouds, and vice versa
      3. Performing ablation studies under different conditions regarding input point clouds (e.g., performance of 20% synthetic, 80% real point clouds)  
   3. Key words: 3D semantic segmentation, state-of-the-art DL networks,  ablation studies, understanding domain gap

Requirements

-       Experience with one of the established, selected DL frameworks (e.g., PyTorch)

-       Basic programming skills in Python (preferable), or similar

-       Basic understanding of 3D point clouds segmentation algorithm