The first part of this work introduces concepts and algorithms in optimization theory which have been used in neural network research. The second covers main neural network models and their theoretical analysis and the third part introduces various neural network models for solving nonlinear programming problems and combinatorial optimization problems.