Abstract: The diversity of object motion and the complexity of background decrease the robustness of object tracking. Similarity of background colors, changes in illumination and object deformation lower the accuracy of the object template and the robustness of object tracking. To deal with this problem, a template update mechanism based on feature-learning and feature-memory was proposed. The algorithm built an object template library by preserving abundant information of the object. By matching the object with the object template library, the state of the object was obtained and the object was then tracked by particle filter. Experimental results show that the proposed method has better accuracy and robustness than the particle filter based on traditional object template update strategies.