Object detection algorithm based on deep learning has achieved great success, significantly better than the effect of traditional algorithms, and even surpassed human in many scenarios. Unlike RGB cameras, infrared cameras can see objects even in the dark, which can be used in many fields like surveillance and autonomous driving. In this paper, a lightweight target detection algorithm for embedded devices is proposed, and the algorithm is accelerated and deployed using Xilinx’s Ultrascale+MPSoC ZU3EG FPGA. The accelerator runs at a 350MHz frequency clock with throughput of 551FPS and power of only 8.4W. As for accuracy, the intersection over union (IoU) of the algorithm achieves an accuracy of 73.6% on FILR datasets. Compared to the previous work, the accelerator design improves performance by 2.59× and reduces power consumption by 2.04×.