Abstract: A new nonlinear blind source separation (NBSS) algorithm is proposed to solve the problem of gearbox composite fault analysis in this paper. In this algorithm, the back propagation (BP) neural network is used to approximate the inverse of the nonlinear aliasing model, and the independent component analysis (ICA) algorithm is performed on the signal processed by the BP neural network; then, the negative entropy of the signal after ICA is taken as the fitness function and genetic algorithm is used to optimize the parameters of the BP neural network; finally, using the optimized BP neural network parameters, the observed aliasing signal is decomposed and the pure vibration source signals are separated. Compared with the kernel independent component analysis (KICA) algorithm optimized by the particle swarm optimization (PSO) algorithm, the separated signals extracted by this method have higher accuracy, which provides the key technology and an effective method for the diagnosis of gearbox composite fault.