This paper is focused on the flying inverted pendulum problem, i.e., how to balance a pendulum on a flying quadrotor. After analyzing the system dynamics, a three loop cascade control strategy is proposed based on active disturbance rejection control (ADRC). Both the pendulum balancing and the trajectory tracking of the flying quadrotor are implemented by using the proposed control strategy. A simulation platform of 3D mechanical systems is deployed to verify the control performance and robustness of the proposed strategy, including a comparison with a Linear Quadratic Controller (LQR). Finally, a real quadrotor is flying with a pendulum to demonstrate the proposed method that can keep the system at equilibrium and show strong robustness against disturbances.