Real-time object tracking is one of the most crucial tasks in the field of computer vision. Many different approaches have been proposed and implemented to track an object in a video sequence. One possible way is to use mean shift algorithm which is considered to be the simplest and satisfactorily efficient method to track objects despite few drawbacks. This paper proposes a different approach to solving two typical issues existing in tracking algorithms like mean shift: (1) adaptively estimating the scale of the object and (2) handling occlusions. The log likelihood function is used to extract object pixels and estimate the scale of the object. The Extreme learning machine is applied to train the radial basis function neural network to search for the object in case of occlusion or local convergence of mean shift. The experimental results show that the proposed algorithm can handle occlusion and estimate object scale effectively with less computational load making it suitable for real-time implementation.