To resolve the “false negative” and “false positive” problems in the current line segment detectors, we proposed a novel line segment detector which organizes pixels under the guidance of pixels’ gradient and improves reliability by step-by-step quality control. Before organizing pixels, initial optimized seed points were extracted according to image gradient. Under organizing, in order to solve the bad performance, e.g. noise of weak gradient, pixels were grouped and connected by considering proximity, orientation and gradient optimization to control false negative. After organizing pixels, chain division and hypothesis testing for quality control were employed to avoid false positive. When compared with state-of-the-art algorithms by visible light and color infrared (CIR) images, such as Probability Hough Transform (PHT), EDlines and LSD (Line Segment Detector), the proposed algorithm has advantages in efficiency and robustness, so it is of great significance for digital photogrammetry, computer vision and remotely sensed information extraction.