X-ray Pulsar Navigation (XPNAV) uses the Time Difference of Arrival (TDOA) of the pulsar signal between the spacecraft and Solar System Barycentre (SSB) to determine position. In this paper, a novel method to improve the performance of XPNAV via exploiting the pulsar position vector is proposed. First, the field of view of the collimator is utilised to find the pulsar orientation direction. Then, a searching strategy based on the modified Powell method under given coordinate frames is proposed. We also mathematically prove the existence of the extreme value of the searching strategy. Subsequently, an observation model based on the pulsar radiation vector is presented and applied to formulate the observation function together with pulsar time transfer function. Finally, an Adaptive Divided Difference Filter (ADDF) algorithm is introduced to iteratively estimate the position and velocity of the spacecraft. Numerical simulations show that the vector searching method is feasible and the pulsar radiation direction can improve the navigation performance by 75%. The simulation results also show that the ADDF performs better than Unscented Kalman Filtering (UKF) and DDF in position estimation.