With a single-photon detector, photon-counting LiDAR (PCL) captures a large amount of background noise along with the target scattered/reflected echo signals, because of the influence of factors such as the background environment, target characteristics, and instrument performance. To accurately extract the signal photons on the ground surface from a noisy photon point cloud (PPC), this paper presents an adaptive denoising approach for PPC using two levels of voxels. First, coarse denoising is performed utilizing large-scale voxels, which are built based on the spatial distribution features of the PPC. The density of the voxel is then used to select the voxels that contained dense signal photons. Second, fine denoising with small-scale voxels is conducted. These voxels are built using the nearest neighbor distance, and a topological relationship between voxels is used to further extract voxels containing signal photons aggregated on the ground surface. Finally, this method is performed on the PPC from ATL03 datasets collected by the Ice, Cloud, and Land Elevation Satellite-2 both during daytime and at night and compared with the improved Density-Based Spatial Clustering of Applications with Noise (DBSCAN), improved Ordering Points to Identify the Clustering Structure (OPTICS), and the method used in the ATL08 datasets. The results show that the proposed method has the best performance, with precision, recall, and F1 score of 0.98, 0.97, and 0.98, respectively.