Different approaches have been proposed over the last few years for improving holistic methods for face recognition. Some of them include color processing, different face representations and image processing techniques to increase robustness against illumination changes. There has been also some research about the combination of different recognition methods, both at the feature and score levels. Embedded hidden Markov model (E-HHM) has been widely used in pattern recognition. The performance of Face recognition by E-HMM heavily depends on the choice of model parameters. In this paper, we propose a discriminating set of multi E-HMMs based face recognition algorithm. Experimental results illustrate that compared with the conventional HMM based face recognition algorithm the proposed method obtain better recognition accuracies and higher generalization ability.