As analytical fusion tracking algorithms based on visible and infrared images always have low robustness in complex environment, a novel adaptive analytical fusion tracking algorithm based on optimized co-training framework was proposed. Firstly, selecting the most discriminative weak classifiers from weak classifier pools based on infrared and visible images respectively are achieved by weighted multiple instance learning boosting technology, which relieving classifiers’ discriminative capacity decreasing owing to the added error positive samples. Then, classifiers’ sample bags are updated by co-training criterion under the help of adaptive prior knowledge import strategy. Lastly, efficiency analysis of the proposed algorithm was achieved based on error model. Comparative experiments on multiple sequences tracking show the contributions for improving tracking robustness from different parts of the proposed algorithm, and then, demonstrate that it outperforms state-of-the-art tracking algorithms based on single source image or other fusion schemes on robustness.