Object Detection using Lidar

State-of-the-art AI Models for processing Lidar data in order to detect objects: Pointnet, Voxelnet, SECOND, and Pointpillar are explained

Yağmur Çiğdem Aktaş


A major breakthrough in recognition and detection tasks on images was due to moving from hand-crafted features to machine-learned features. PointNet the first, an end-to-end deep neural network that learns point-wise features directly from point clouds brought the same breakthrough for point cloud data and after this new architecture, various types of models are proposed to perform object classification, detection, and segmentation tasks on the point cloud.

In this post, I will explain the PointNet and the models coming out to accomplish more complicated tasks with Lidar thanks to the usage of Pointnet embedded in their architectures.

First, let’s take a look at the challenges of processing Lidar data and how PointNet solves these problems for the first time:

Challenges for processing point clouds :

  • The lidar data is unstructured so the model should be permutation invariant (if you have n points you have n! permutations of ordering and processing this data). In other words, in contrary to the images where we can go from left to right and from top to bottom by saying “this is the first pixel, this is the second pixel…”, in lidar data, we don’t have any exact order of point, we don’t have any start or end point that model will follow. So, each time we run the model for a train or test process, the model should be robust enough to give the same result independently from the order of processing the points.
Unordered lidar data
  • Interaction of points (the relation between neighbor points (local features) and general relation (global features) should be extracted. For classification, the global features are important since the task is to decide on a global class for the whole point cloud. For segmentation, a combination of local and global knowledge is required
  • The model should be transform-invariant (translation — rotation). A chair is still a chair when it's rotated 90 degrees, etc.

PointNET — 2016