The particle filter is an effective technique for the state estimation in non-linear and non-Gaussian dynamic systems. A novel method for infrared object robust tracking based on particle filters was proposed. Under the theory framework of particle filters, the posterior distribution of the infrared object is approximated by a set of weighted samples, while infrared object tracking is implemented by the Bayesian propagation of the sample set. The state transition model is chosen as the simple second-order auto-regressive model, and the system noise variance is adaptively determined in infrared object tracking. Infrared objects are represented by the intensity distribution, which is defined by the kernel-based density estimation. By calculating the Bhattacharyya distance between the object reference distribution and the object sample distribution, the observation probability model is constructed. Experimental results show that our method is effective and steady.