A novel approach to estimate the real-time moving trajectory of an object is proposed in this paper. The object's position is obtained from the image data of a charge coupled device (CCD) camera, while a state estimator predicts the linear and angular velocities of the moving object. To overcome the uncertainties and noises residing in the input data, a Kalman filter and neural networks are utilized cooperatively. Since the Kalman filter needs to approximate a nonlinear system into a linear model in order to estimate the states, there still exist errors as well as uncertainties. To resolve this problem, in this approach, the Kohonen networks, which have a high adaptability to the memory of the input–output relationship, are utilized for the nonlinear region. In addition to this, the Kohonen network, as a sort of neural network, can effectively adapt to the dynamic variations and become robust against noises. This approach is derived from the observation that the Kohonen network is a type of a self-organized map and is spatially oriented, which makes it suitable for determining the trajectories of moving objects. The superiority of the proposed algorithm compared with the extended Kalman filter is demonstrated through real experiments.