Midcourse ballistic target, releasing warhead and large numbers of decoys and forming dense cluster, presents a new challenge for target tracking with spacebased infrared focal plane array ( IRFPA ). The numbers of the identified target vary because of the finite resolution of IRFPA. The target trajectory on IRFPA shows strong nonlinear characteristic. To cope with these problems, a tracking algorithm based on random finite set was proposed. Random finite set is a theoretically unified framework of optimal Bayesian multitarget tracking filter, but the recursion of posterior joint multitarget distribution is not practical in use due to the computational hurdle. Probability Hypothesis Density(PHD) is the first moment of multitarget posterior distribution, and the PHD filter is the suboptimal and practical alternative within the framework of random finite set. Sequential Monte Carlo method was proposed to propagate PHD. The target quantity was estimated by summing up all particle's weight. The kmeans method was adopted to cluster PHD to estimate the target states. Scenario simulation was set up, where the algorithm was tested under the conditions of different handover tracking tasks，false alarm rates and target numbers. Simulation results show that, the algorithm can simultaneously track the numbers and states of the targets among midcourse ballistic target group on spacebased IRFPA. 