According to the basic principle of infrared photon detectors, the basic physical mechanism limitation of photon detector performance was analyzed. The normalized detectivity which depended on the photon noise of background radiation was deduced. The change of the normalized detectivity under background limitation with the cutoff wavelength and background temperature was calculated theoretically. The calculation results were analyzed theoretically and the corresponding conclusion was given. Finally, a method for improving the normal detectivity under background limitation was proposed.