This paper addresses robust stability and position tracking problems in teleoperation systems subject to varying delay in the communication medium, uncertainties in the models of manipulators, and non-passive interaction forces in the terminations. Fixed-structure nonlinear control law is developed based on the notion of Interconnection and Damping Assignment Passivity-Based Control (IDA-PBC) scheme. Then, utilizing the Lyapunov–Krasovskii theorem, sufficient conditions are derived in terms of Linear Matrix Inequalities (LMIs) to tune the controller parameters. Differently from literature, the objectives are achieved without requirement for any passive parts in the model of interaction forces. Comparative simulations and experimental results demonstrate the applicability and superiority of the proposed method.