Autonomous control design of an unmanned aerial manipulator for contact inspection

Abstract

In recent years, autonomous control based on contact inspections in unknown environments is a new hot and difficult point in robotics research. This paper presents a new control law for unmanned aerial manipulator (UAM) to perform contact inspection tasks on vertical surfaces. The selected circular image feature decouples the position and attitude of UAM, so an image-based impedance control is proposed to control the position and track the contact force. The developed controller uses geometric methods to control the attitude. In addition, the designed aerial manipulator decouples the roll and pitch of the UAV from the UAV, which improves the system’s stability. Experiments have been carried out to demonstrate the feasibility of this method.

1. Introduction

In recent years, the use of unmanned aerial vehicles (UAVs) especially rotary-wing unmanned aerial manipulators (UAM) in aerial inspections has shown a significant increase, which made the great breakthroughs and progress. UAM consisting of an UAV and active operating mechanism can complete some simple tasks currently, such as grasping [1, 2], sampling [3], transportation [4–6], contact inspection [7–9] and sensor installation [10]. In addition, UAM has great potential applications in inspection tasks such as large-scale infrastructure inspections that are difficult for humans to reach. It is widely used in the field of nuclear reactor containment structure inspection, oil and gas pipeline leak detection and maintenance [11], and large bridge pier structure inspection and maintenance. These large public infrastructures require regular maintenance to prevent leaks, breakages, and disasters. Human inspections are limited by inefficiency and high safety risks. At the same time, due to the excessive labor intensity of high-altitude operations (such as wind turbines, poles, and towers), the time of each operation is strictly limited. The UAV platform has the characteristics of high mobility, hovering in the air, and low cost. It has great development space and prospects for carrying out operating mechanisms. These aerial vehicles must be capable of environment perception and autonomous operation, including dexterous manipulators, cameras, force sensors, and distance sensors. Therefore, UAM composed of an UAV-carrying manipulators has a very wide application prospect and can provide flexible operation capability

UAM’s ability to perceive the outside world determines its autonomous control ability. At present, most research on UAM autonomous control uses visual servo control [; 12, 13]. The current mainstream visual servo control algorithms are mainly divided into a position-based visual servo (PBVS) and image-based visual servo (IBVS). Compared with IBVS, PBVS needs to use image features to calculate the position information, which requires a large amount of calculation. IBVS can directly obtain position information through images, with higher control robustness and less calculation amount, which is especially suitable for scenarios that require stable interaction between UAM and the external environment. Meanwhile, since the external environment is usually unknown, under limited perception, some studies use image moments [14–16] and points [17] as visual features to design controllers. In ref. [18], a PBVS-based geometric attitude controller was designed to solve the dynamic landing problem of the position mobile platform.

When interacting with the external physical environment, maintaining the stability of the strongly coupled system composed of UAM and the external environment is a very important and worthwhile research field. Active operating mechanisms are also required for inspection tasks that interact with the external environment. Ref. [19] used a fully driven UAV and a two DOF lightweight aerial manipulator for continuous weld inspection on curved surfaces. Ref. [20] designed the Delta manipulator operating mechanism according to tasks such as foreign body removal in transmission lines and repair of large-scale pipeline welds, which have a high dynamic response and can dynamically compensate for changes in the UAV’s attitude to achieve the effect of a self-stabilizing gimbal. In addition, position-based impedance control is also a common solution. Ref. [21] designed an impedance controller based on a PID model to improve stability. Ref. [22] designed an experimental scenario for a glass cleaning task to demonstrate the interaction between UAM and contact force and designed a hybrid force/motion controller based on a spring-mass-damper second-order impedance model. At present, the research mainly uses the quadrotor aircraft as the flight platform, and the quadrotor aircraft needs to change its attitude and move due to its underactuated characteristics. However, in the past few years, full-drive aerial platforms have emerged to address the above issues. Ref. [21] designed a full-drive UAV platform with tilt rotors, which overcomes the requirement of the UAV to change attitude while moving. Furthermore, ref. [22] designed a manipulator with impedance control and proposed a new normal estimation controller to improve the stability of contact inspection and verify the effectiveness. This kind of active robot has many applications at present [23–28].

Figure 1. Unmanned aerial manipulator in contact, which equipped with onboard computer and an NDT sensor attached on the surface.

In this paper, we consider designing a UAM for contact inspection. At the same time, we designed a 2-DOF manipulator for contact inspection, decoupling the UAV attitude from the end effector and overcoming the insufficient drive characteristics of traditional quadrotors and the instability of visual servo. Then, we describe the constant contact force algorithm design of the manipulator. Next, we briefly describe the designed visual servo-based autonomous controller. Finally, the designed UAM autonomous control is tested and analyzed through experiments. The design of the UAM is shown in Fig. 1. The main contributions of this paper can be summarized as follows:

1. 1. For the contact inspection scenario, a controller that decouples the position and attitude of the UAM is proposed. The controller adopts a cascade control, in which the outer loop adopts IBVS control. Specifically, we proposed a one-dimensional image-based impedance control model, which improves the stability of UAM in the process of interaction with the external environment. The inner loop is a geometric attitude controller, which effectively solves the risk of loss of control caused by the strong coupling of UAM by decoupling the position and attitude. The stability of the proposed controller is verified in experiments.

2. 2. A working mechanism for contact operation is designed, which decouples the attitude from the UAV flight platform, simplifies the control design, and improves the stability.

In this paper, we consider designing a UAM for contact inspection and take NDT as an example. Meanwhile, we designed a 2-DOF manipulator to decouple the UAV attitude from the end effector, overcoming the underactuated characteristics of traditional quadrotors and the instability of visual servo. Then, we described the constant contact force algorithm design of the manipulator. Next, we give a brief description of the designed visual servo-based autonomous controller. Finally, the designed UAM autonomous control is tested and analyzed by experiments. The design of the UAM is shown in Fig. 1.

The rest of this paper is organized as follows. The second part introduces the system design and operation flow of UAM, and the third part introduces the system modeling and controller design. Then, Section 4 shows the experimental results verifying the effectiveness of the proposed method. Finally, Section 5 concludes the article.

2. System design

2.1. Mechanism design

Figure 2 shows the overall structure and schematic diagram of the NDT inspection process. The proposed UAM is mainly composed of a quadrotor, a 2-DOF manipulator, and an NDT mechanism. It has the following advantages: First, the objects to be detected are mostly large-scale infrastructures, and the shape of the objects is vertically distributed. Detection from the open environment on the side can prevent the UAM from being affected by collision accidents. Second, the two joints of the designed manipulator can compensate for the attitude disturbance of the UAV during the contact inspection and improve the stability of the inspection as well. Third, the designed NDT mechanism can complete the inspection quickly and effectively without damaging the object.

Figure 2. Schematic diagram of the NDT inspection process.

2.2. Contact inspection mechanism design

The designed manipulator overview is shown in Fig. 3. The manipulator has three joints, two of which are installed at the bottom to control the roll and pitch angles of the manipulator to compensate for the attitude disturbance of the UAV. The third joint is installed behind the manipulator and is used to control the forward and backward movement of the NDT mechanism relative to the manipulator. The robotic arm is made of carbon fiber tubes to reduce weight, and the NDT mechanism is installed at the end of the manipulator to reciprocate with the third joint of the manipulator. It is mainly composed of an elastic compliance mechanism, limit pin, fixed seat, annular stabilizer, and probe. Among them, the elastic compliance mechanism can ensure the safety of the UAM in contact with the detected object, provide active compliance control error, and design a control algorithm for constant contact force.

Figure 3. UAM manipulator design overview.

Since the modeling in the contact mode is too complicated, the model is modeled on a two-dimensional plane perpendicular to the UAM contact inspection. Ignoring the friction force of the manipulator, define the function:

(1) \begin{align} {{f}_{c}}={{f}_{e}} + {{f}_{k}}-{{f}_{s}} \end{align}

(2) \begin{align} {{f}_{k}}={{\lambda }_{k}}({{x}_{o}}-{{x}_{i}}) \end{align}

(3) \begin{align} {{f}_{s}}={{\lambda }_{s}}({{\theta }_{o}}-{{\theta }_{i}}) \end{align}

where ${f}_{c}$ denotes the contact force between the end effector and object, to simplify the model, we discuss the one-dimensional model of the contact force with respect to the normal direction of the object in this paper. ${f}_{k}$ and ${f}_{s}$ denote the elastic force and pulling force of the steel wire drive motor, respectively. ${f}_{e}$ denotes the contact force perpendicular to the contact surface generated by the annular stabilizing structure, and the direction is perpendicular to the contact surface and inward, ${f}_{e}$ is a constant value during the contact work. ${x}_{o}$ , ${x}_{i}$ denotes the initial length and actual length of the spring, respectively.

${\theta }_{o}$ and ${\theta }_{i}$ are the absolute angles corresponding to the wire spin rudder at the spring length of ${x}_{o}$ and ${x}_{i}$ , respectively, and ${\lambda }_{k}$ , ${\lambda }_{s}$ is the gain factor. Let $r$ be the radius of the wire spin rudder, we have

(4) \begin{align} ({{x}_{i}}-{{x}_{o}})=\frac{\pi r}{180}({{\theta }_{i}}-{{\theta }_{o}}) \end{align}

Using (1, 2, 3, 4), the contact force can be written as:

(5) \begin{equation} \begin{split}{{f}_{c}}&=({{\lambda }_{k}}{{k}_{i}}-{{\lambda }_{s}})\widetilde{\theta }+{{f}_{e}} \\[5pt] &={{J}_{c}}\widetilde{\theta }+{{f}_{e}} \end{split} \end{equation}

where ${{k}_{i}}=\dfrac{\pi r}{180}$ , $\widetilde{\theta } = ({{\theta }_{i}}-{{\theta }_{o}})$ and ${{J}_{c}} = {{\lambda }_{k}}{{k}_{i}}-{{\lambda }_{s}}$ . The force from diagram of UAM is shown in Fig. 4.

Figure 4. NDT mechanism contacting to target surface with contact force.

2.3. Onboard computer and sensors

The flight control unit (FCU) is an open-source FCU for pixhawk with PX4 firmware, which can stably control the pose of the UAM during a flight mission. The FCU provides an “Offboard” flight mode for position and attitude control by an external onboard computer. These instructions are provided by the Jetson Nano onboard computer. The camera sensor is equipped with an Intel RealSense D435 depth camera, which feeds the captured visual features to the onboard computer for algorithmic processing.

3. Autonomous control

When performing contact inspections, the pixhawk FCU onboard the UAM flies in multiple modes (programmed control, position control, etc.). The UAM is set up to take off and land via the remote control, and switch flight modes as the mission requires. Researchers can remotely control the UAM to complete NDT tasks through remote control. First, take off the UAM near the desired position through the position mode, switch the FCU to Offboard mode, and position the UAM directly in front of the detection target position. The UAM moves to the target under program control and touches the end effector to the target surface. Maintain a certain tilt angle during contact to provide contact force, leave the surface, and return to a safe position after completing the detection and switch back to manual mode through the remote control and land. The state diagram of the NDT task is shown in Fig. 5.

Figure 5. State flowchart diagram for autonomous contact inspection task.

3.1. Inspection process

Autonomous NDT control consists of the following sub-tasks, the switching between each task state is triggered by the corresponding event (such as when the target appears in the camera field of view), and set an error threshold (error per direction in ZYX) during task switching, allowing the UAM to stay within the error threshold. The operator can switch the FCU to the auto-stabilized mode to keep hovering at any time through the button on the RC or activating the emergency safety switch as a safety protection measure. Figure 4 shows the state flow diagram for autonomous NDT control.

1. (1) Move to a location near the target: The position of the target is set to be unknown, so first set an initial state position within 1−1.5 m of the target, and control the UAM to fly to the vicinity of the target surface to ensure that the target object that needs to be non-destructively detected is within the field of view of the camera. In the experiment, UAM’s position and attitude information is defined and provided by the Optitrack motion capture system and can be defined by GPS coordinates in applications. The operator directly controls the sightline to fly to the set position and switches the FCU flight mode to Offboard mode through RC.

2. (2) Pose calibration: When the mode is switched to autonomous control and the target appears within the recognizable field of view of the camera ( $\le2\;\text{m}$ ), the UAM will switch to the pose calibration state after a buffer time (5 s), in which the visual positioning data from the depth camera and the attitude estimation data fusion of the UAV calibrate the position and attitude of the UAM, and the visual tracking algorithm ensures that the front axis of the manipulator (NDT mechanism, that is, the axis of the manipulator) is vertically aligned with the target and controls the UAM to contact the target. The onboard computer has a high refresh rate (70 Hz) to process the attitude estimation data, the data error of the manipulator attitude estimation caused by the time delay is within the allowable threshold. The designed coordinate points will eventually control the NDT mechanism to be perpendicular to the target surface and level with the ground.

3. (3) Return to the safe position after completing the NDT inspection: After the pose calibration is completed, the NDT mechanism is already in front of the target, and the UAM switches to the APPROCHING_TARGEET state. The designed controller generates a control trajectory by calculating the image error between the current visual circle image moment feature and the desired feature, the UAM fuselage moves toward the axis (facing the target) and moves close to the target. When the contact force generated by contact between the end effector and the target exceeds the set threshold, the UAM switches to NDT_MANIPULATION mode. At this time, the UAV keeps the pitch angle forward to continue to provide the contact force, ensuring that the NDT mechanism is in close contact with the target. During this period, the two joints of the manipulator make angle compensation for the attitude deviation of the UAV, keeping the vertical contact between the NDT mechanism and the target and the contact force constant. After the detection is completed, the UAM switches to the WAITING_FOR_COMMAND mode. The UAM falls back to a safe position and remains to hover until the operator switches back to manual mode via the RC.

3.2. System modeling

We use the coordinate system defined in ref. [14] as the frame in our proposed autonomous control algorithm, the equations of the dynamics with UAM can be depicted as follows:

(6) \begin{align} \dot{\textbf{p}}={\textbf{v}} \end{align}

(7) \begin{align} m\dot{{\textbf{v}}}=m{g{\textbf{e}}_{3}}+{{f}_{c}}-f{\textbf{R}}{{\textbf{e}}}_{3} \end{align}

(8) \begin{align} \dot{{\textbf{R}}}={\textbf{R}}{{\Omega }^{\wedge }} \end{align}

(9) \begin{align} {\textbf{J}}\dot{\boldsymbol{\Omega } }\,{+}\, \boldsymbol{\Omega } \times {\textbf{J}}\boldsymbol{\Omega } \, = \, \boldsymbol{\tau } \end{align}

where $\textbf{p}={{ [ x,y,z ]}^{\text{T}}}$ and ${\textbf{v}}={{ [{{v}_{x}},{{v}_{y}},{{v}_{z}} ]}^{\text{T}}}$ are the position and velocity vectors of the UAM’s center of mass relative to the inertial frame, respectively. ${\textbf{R}}\in SO(3)$ denotes the transformation matrix form body-fixed frame ${\sum }_{b}$ to inertial frame ${\sum }_{i}$ , ${{\textbf{e}}_{3}} = {{ [ 0,0,1 ]}^{T}}$ and the hat map $(\hat{\cdot }):{{\mathbb{R}}^{3}}\to SO(3)$ is the condition that ${\textbf{a}^{\wedge }}{\textbf{b}}={\textbf{a}}\times {\textbf{b}}$ for all ${\textbf{a}},{\textbf{b}}\in{{\mathbb{R}}^{3}}$ . $\boldsymbol{\Omega }$ is the angular velocity in body-fixed frame ${\sum }_{b}$ to inertial frame ${\sum }_{i}$ . $f$ denotes the thrust force of UAM, and $mg$ is the gravity. $\boldsymbol{\tau } = {{[{{\tau }_{1}},{{\tau }_{2}},{{\tau }_{3}}]}^{T}}$ represent the resultant moment of thrust force and contact force in the body-fixed frame.

3.3. Image features and system dynamics

In the contact manipulation task, a camera is installed parallel to the aerial manipulator. For the camera model, we place a black circular pattern next to the detection target to locate the UAM to complete the NDT task. In this paper, we use the classical camera model of a pinhole. For each observed feature point can be described as ${{\textbf{P}}_{c}} = {{ [{{X}_{c}},{{Y}_{c}},{{Z}_{c}} ]}^{T}}$ of the camera frame ${{\Sigma }_{c}}\;:\;{{O}_{c}}-{{x}_{c}}{{y}_{c}}{{z}_{c}}$ can be projected onto the normalized image plane as:

(10) \begin{equation}{\textbf{p}_{i}}={{[{{x}_{i}},{{y}_{i}}]}^{\text{T}}}={{\left [ \frac{{{X}_{i}}}{{{Z}_{i}}},\frac{{{Y}_{i}}}{{{Z}_{i}}} \right ]}^{\text{T}}} \end{equation}

Ref. [29] has proposed a virtual camera approach to deal with the uncontrollable roll and pitch of UAM, the roll and pitch of the quadrotor are decoupled from the aerial manipulator during the contact manipulation; in this paper, we define a virtual camera frame $ \{{{\sum }_{{{c}^{*}}}}\;:\;{{O}_{{{c}^{*}}}}-{{x}_{{{c}^{*}}}}{{y}_{{{c}^{*}}}}{{z}_{{{c}^{*}}}} \}$ and foreign object frame $ \{{{\sum }_{t}}\;:\;{{O}_{t}}-{{x}_{t}}{{y}_{t}}{{z}_{t}} \}$ . The origin of the virtual camera frame ${\Sigma }_{{{c}^{*}}}$ is the same as the real camera frame ${\Sigma }_{c}$ . The coordinate of the observed points in ${\sum }_{{{c}^{*}}}$ are represented by ${{\textbf{P}}_{{{c}^{*}}}}={{ [{{X}_{{{c}^{*}}}},{{Y}_{{{c}^{*}}}},{{Z}_{{{c}^{*}}}} ]}^{\text{T}}}$ and denote ${\textbf{R}}_{c}^{{{c}^{*}}}$ be the rotation matrix of ${\sum }_{{{c}^{*}}}$ with respect to ${\sum }_{c}$ , and then, we have ${{\textbf{P}}_{{{c}^{*}}}}={\textbf{R}}_{c}^{{{c}^{*}}}{{P}_{c}}$ and the corresponding image coordinate ${\textbf{p}_{{{i}^{*}}}}={{[{{x}_{{{i}^{*}}}},{{y}_{i*}}]}^{\text{T}}}$ .

Figure 6. Block diagram of the designed autonomous control strategy, in which $\frac{1}{M_{s}s^{2}+D_{s}s+K_{s}}$ is the Laplace transformation of the image-based impedance control scheme.

We choose to consider a circle feature attached to the object surface so that the j + k order image moments ${m}_{j,k}$ is defined as:

(11) \begin{equation}{{m}_{j,k}}=\int{\int{x_{{{i}^{*}}}^{j}y_{{{i}^{*}}}^{k}d{{x}_{{{i}^{*}}}}}}d{{y}_{{{i}^{*}}}} \end{equation}

the contour centroid $(\tilde{x},\tilde{y})$ of ${m}_{j,k}$ is given by $\tilde{x} = \dfrac{{{m}_{10}}}{{{m}_{00}}}$ and $h{\tilde{y}} = \dfrac{{{m}_{01}}}{{{m}_{00}}}$ , where ${m}_{00}$ denotes the area of the circular outline; therefore, the image moment features can be defined as

(12) \begin{align} {{\textbf{s}}_{t}}=[{{s}_{x}},{{s}_{y}},{{s}_{z}}] \end{align}

(13) \begin{align} {{s}_{x}}=\tilde{m}\tilde{x},\, \,{{s}_{y}}=\tilde{m}\tilde{y},\, \,{{s}_{z}}=\tilde{m} \end{align}

(14) \begin{align} \tilde{m}=\sqrt{m_{00}^{*}/{{m}_{00}}} \end{align}

According to the selected circular image moment feature, ${s}_{z}$ can be expressed by:

(15) \begin{equation}{{s}_{z}}=\sqrt{m_{00}^{*}/{{m}_{00}}}={{Z}_{c}} \end{equation}

Let ${Z}_{{{v}^{*}}}$ denotes the desired distance between the observed object plane of the camera and the image plane. $m_{00}^{*}$ is the desired value of $m_{00}$ when the desired value ${Z}_{{{v}^{*}}}$ is reached. The relationship between the first derivative of the image feature vector $\dot{\textbf{s}}_{\textbf{t}}$ and the camera velocity $\textbf{v}_{\textbf{c}}^{\textbf{c}}$ relative to the camera frame ${\Sigma }_{c}$ can be expressed as follows:

(16) \begin{equation} {{\dot{\textbf{s}}_{t}}}=-\frac{1}{{{Z}_{{{v}^{*}}}}}\textbf{v}_{\textbf{c}}^{\textbf{c}} \end{equation}

3.4. Visual servo control

The error of the image feature is defined as follows formula:

(17) \begin{equation}{{\textbf{e}}_{t}}={{\textbf{s}}_{t}}-{\textbf{s}}_{t}^{*} \end{equation}

where ${\textbf{s}}_{t}^{*}$ is the desired image feature vector.

The contribution of (16) is that decoupled the UAM’s translation and rotation, in which the model can be simplified and designed with controllers for each transformation.

The purpose of the designed visual servo controller is to stabilize the contact force ${f}_{c}$ in terms of the force error. The next subsection will demonstrate the linear relationship between the UAM’s position and image features, which shows the further influence of image features on contact force ${f}_{c}$ . The computation of desired image feature ${\textbf{s}}_{t}^{*}$ will be elucidated in the next subsection as well. Then, the visual servo control law is designed as:

(18) \begin{equation} {\textbf{v}}_{c}^{c}={\textbf{L}}_{s}^{-1}{{\boldsymbol{\eta } }_{c}}{{\textbf{e}}_{t}} \end{equation}

where ${{\textbf{L}}_{s}}\in{{\mathbb{R}}^{3\times 3}}$ is the first three columns of the interaction matrix illustrated in ref. [18], ${\boldsymbol{\eta } }_{c}$ is the $3\times 3$ positive diagonal matrix.

3.5. IBVS controller design for inner loop

In this part, image-based impedance control is proposed for contact force tracking and control of the force impedance of UAM along the normal direction to the surface of the observed object (the $z$ axis direction of ${f}_{c}$ ). According to the proposed decoupling attribute image feature ${\textbf{s}}_{t}^{*}$ , the image-based impedance control considered only controls the translational DOFs of UAM in this subsection. The impedance model equation is considered as follows:

(19) \begin{equation}{{M}_{s}}({{\dot{v}}_{z}}-\dot{v}_{z}^{*})+{{D}_{s}}({{v}_{z}}-v_{z}^{*})+{{K}_{s}}(z-{{z}^{*}})=\Delta{{f}_{c}} \end{equation}

where ${v}_{z}^{*}$ and ${z}^{*}$ are the desired value of ${v}_{z}$ and $z$ , respectively. ${M}_{s}$ , ${D}_{s}$ , ${K}_{s}$ are the desired inertia, damping, and stiffness coefficients of the system with constant numbers. $\Delta{{f}_{c}}={{f}_{c}}-f_{c}^{*}$ , where $f_{c}^{*}$ is the desired contact force w.r.t $\;{f}_{c}$ . Here we address the compliance problem of contact force (considered as an external disturbance) when UAM performs NDT by designing an impedance controller as a function.

Meanwhile, let ${\textbf{L}}^{e}$ and ${\textbf{L}}^{c}$ denote the relative position between the object with respect to the end effector and camera, respectively, and then, we can get the following relation about the variation of ${\textbf{L}}^{e}$ as:

(20) \begin{equation} \Delta {\textbf{L}}^{e}={\textbf{T}}^{e} \Delta {\textbf{L}}^{c} \end{equation}

where ${\textbf{T}}^{e}$ is the transformation matrix from camera frame ${\sum }_{c}$ to end-effector frame ${\sum }_{e}$ . $\Delta {\textbf{L}}^{e}$ , $\Delta {\textbf{L}}^{c}$ are the variation of ${\textbf{L}}^{e}$ , and ${\textbf{L}}^{c}$ , respectively. It is worth noting that the installation direction of the camera’s optical axis is parallel to the aerial manipulator, and then, we can get $\Delta{{\textbf{L}}^{e}}{{\textbf{e}}_{3}}=\Delta{{\textbf{L}}^{c}{{\textbf{e}}_{3}}}=-\Delta{{s}_{z}}$ , where $\Delta{{s}_{z}}=s_{z}^{*}-s_{z}^{i}$ , where $s_{z}^{i}$ denotes the input image feature with the onboard computer. Substitute it in (19), the visual impedance control equation can be expressed as:

(21) \begin{equation}{{M}_{s}}\Delta{{\ddot{s}}_{z}}+{{D}_{s}}\Delta{{\dot{s}}_{z}}+{{K}_{s}}{{s}_{z}}=\Delta{{f}_{c}} \end{equation}

The block diagram of the designed control strategy shown in Fig. 6. Recall that the image feature vector ${s}_{t}$ is only related to the translation transformation of the UAM; in this part, we control the rotational transformation of the UAM through the proposed geometric controller. Let the tracking errors of position and velocity be given by:

(22) \begin{align} {{\textbf{e}}_{x}}=\textbf{p}-{\textbf{p}_{d}} \end{align}

(23) \begin{align} {{\textbf{e}}_{v}}={\textbf{v}}-{\textbf{v}}_{c}^{c} \end{align}

so the control output ${\textbf{f}}$ can be written as follows:

(24) \begin{equation} {\textbf{f}}=-(-{{k}_{x}}{{\textbf{e}}_{x}}-{{k}_{v}}{{\textbf{e}}_{v}}-mg{{\textbf{e}}_{3}}-{{f}_{c}}+m{\ddot{\textbf{p}_{d}}}) \end{equation}

where ${k}_{x}$ , ${k}_{v}$ are several positive constants.

3.6. Geometric attitude control for outer loop

We consider the desired attitude and angular velocity as ${\textbf{R}}_{d}$ and ${\boldsymbol{\Omega } }_{d}$ , respectively, Let ${{\textbf{R}}_{d}}= [{{\textbf{u}}_{1d}},{{\textbf{u}}_{2d}},{{\textbf{u}}_{3d}} ]$ , in which

(25) \begin{align} \begin{cases}{{\textbf{u}}_{1d}}={{\textbf{u}}_{2d}}\times{{\textbf{u}}_{3d}} \\[7pt]{{\textbf{u}}_{2d}}=\dfrac{{{\textbf{u}}_{3d}}\times \beta }{\left \|{{\textbf{u}}_{3d}}\times \beta \right \|} \\[10pt]{{\textbf{u}}_{3d}}=\dfrac{{\textbf{f}}}{\left \| {\textbf{f}} \right \|} \end{cases} \end{align}

let $\beta ={{ [ \cos{{\psi }^{*}},\cos{{\psi }^{*}},0 ]}^{\text{T}}}$ . Then, we define the error function:

(26) \begin{equation} \mathfrak{S}({\textbf{R}},{{\textbf{R}}_{d}})=\frac{1}{2}tr\left [ I-{\textbf{R}}_{d}^{T}{\textbf{R}} \right ] \end{equation}

Furthermore, we get the angular velocity and attitude error as follows:

(27) \begin{align} {{\textbf{e}}_{\Omega }}=\boldsymbol{\Omega } -\textbf{R}^{T}{{\textbf{R}}_{d}}{\boldsymbol{\Omega }_{d}} \end{align}

(28) \begin{align} {{\textbf{e}}_{R}}=\frac{1}{2}{{({\textbf{R}}_{d}}^{T}{\textbf{R}}-{{\textbf{R}}}^{T}{{\textbf{R}}_{d}})}^{\vee } \end{align}

where the vee map $(\overset{\vee }{\mathop{\cdot }}\,)$ : $SO(3)\to{{\mathbb{R}}^{3}}$ indicates the inverse of the hat map, and define ${k}_{R}$ , ${k}_{\Omega }$ as the positive constant; thus, the control input $\boldsymbol{\tau }$ is obtained as follows:

(29) \begin{equation} \begin{aligned} \boldsymbol{\tau } &=-{{k}_{R}}{{\textbf{e}}_{R}}-{{k}_{\Omega }}{{\textbf{e}}_{\Omega }}+\boldsymbol{\Omega } \times \textbf{J}\boldsymbol{\Omega } \\[5pt] & \quad-{\textbf{J}}\left ( \hat{\boldsymbol{\Omega } }{{\textbf{R}}^{T}}{\textbf{R}}_{d}{{\boldsymbol{\Omega } }_{d}}-{{\textbf{R}}^{T}}{\textbf{R}_{d}}{\dot{\boldsymbol{\Omega } }_{d}} \right ) \end{aligned} \end{equation}

4. Experimental results

This section introduces NDT experiments on vertical surfaces, which can prove the effectiveness of our proposed autonomous control strategy. We will introduce the experimental environment and system configuration and make some analysis of the experimental results. During the autonomous flight of UAM, the effectiveness of NDT depends on the designed autonomous control algorithm and the stability of the airframe.

4.1. Experimental setup

The UAM we used was developed based on an existing quadrotor model (P-600) with a wheelbase of 600 mm; the NVIDIA Jetson Nano onboard computer was selected for real-time calculation of contact force and visual feedback, which runs on Ubuntu 18.04 + ROS Melodic operating system. The flight controller used is Pixhawk 4 (firmware version V1.10).

A 1-DOF manipulator is installed on the top of the quadrotor, connected with a 2-DOF rotary joint, and is driven by three servos (Dynamixel XM-430-W350-T). The total length of the manipulator is composed of three carbon fiber tubes with a total length of 90 cm and extends to the outside of the rotor for contact inspection. an NDT device composed of a ring stabilizer and a measuring probe is installed at the end of the manipulator. For the power system of UAM, we choose DJI TB48D (5700 mAh) battery to provide power for the whole system, and considering the balance of the center of gravity, the battery is installed behind the UAM to offset the weight of the manipulator.

The Intel RealSense D435 camera used to obtain visual feedback information is installed on the connecting rod of the manipulator, and the control current of the manipulator can be directly obtained from the servo to measure the actual contact force.

For the experimental scene, a wooden board (25 $\times$ 55 cm) in the room is chosen to be fixed vertically on the horizontal plane as the interaction plane between the UAM and the environment. The use of wooden boards simulates scenarios where non-destructive testing is limited in practical applications (such as non-destructive testing of oil and gas pipeline surfaces), and we can also assume that the experimental scenario is suitable for large-scale infrastructure testing (such as large-scale wind turbine blade flaw detection), etc. A circular marker (10 cm in diameter) used as a visual image feature was installed above the wooden board as a visual guide. We use external position information from the Optitrack motion capture system and attitude information from the Pixhawk 4 IMU for data fusion. The pose of the plank is unknown, and the initial position and pose of the UAM are arbitrarily random so that the circular marker is always in the camera’s field of view.

The total mass of UAM is 3.5 kg, and the parameters of controller is set as: ${{k}_{x}}=25$ , ${{k}_{v}}=4.3$ , ${{k}_{R}}=13$ , ${{k}_{\Omega }}=1.5$ . Additionally, the gain factors of (1)−(3) are ${{\lambda }_{k}}=0.576$ , ${{\lambda }_{s}}=7.48$ . Figure 7 shows the whole contact inspection process. The desired contact force in maintaining contact between the UAM and wooden board is set as $2.5\;\text{N}$ . UAM first take off to a random position and stably detected the circular image feature. Then, the UAM end effector was driven to a stable position perpendicular to the plane of the wooden board under the guidance of visual servo; after that, UAM maintains the force of $2.5\;\text{N}$ . The system remains free-flying from t = 0−14 s and stays in contact after 14 s.

Figure 7. Snapshots of experiments. The three sub-images on the left (a, b, c) are from a distant top view, and the three sub-images on the right (d, e, f) are from a side view. The three sub-pictures correspond to the initial state of the UAM, the UAM end effector is calibrated in front of the plank, and the UAM is in stable contact with the plank.

4.2. Experimental performance

Figure 8 shows the real-time experimental data of the aerial manipulator contacting the wooden board during the inspection process. Figure 8(a) is the position trajectory of the UAM, (b)−(c) are the velocity trajectories of the UAM, (d)−(e) represent the attitude tracking error of the UAM and the actual and desired contact force between the end effector and the board, respectively. In the approaching stage, the UAM was in free flight with $F_{c}=0$ . After the aerial manipulator touches the wooden board, the UAV flight platform tilts forward to generate contact force. Before t = 14 s, UAM did not contact the board with $F_{c}=0$ . According to the information obtained in the experiment, UAM has no information that contact force can be generated at any time, so the change of $\Delta{{s}_{z}}$ cannot be too fast. According to the image-based impedance control, the UAM took about 5 s to maintain the contact force of the aerial manipulator at around 2.5 N. The designed impedance control law keeps the $F_{c}$ within a certain range, which is still an acceptable error range for UAM in contact inspection. Therefore, the experimental results are acceptable, and the disturbance due to the interaction between the UAM and the environment is still unavoidable for our designed control scheme.

Figure 8. Experiment result of UAM contact inspection with a desired force of $2.5\;\text{N}$ in outdoor scene, the image feature markers were chosen as circular shape on a bridge pier (25 $\times$ 55 cm). (a) Position trajectory of UAM platform in inspection task. (b) Angular velocity of UAM platform in inspection task. (c) Linear velocity of UAM platform in inspection task. (d) Attitude tracking error of geometric controller, (e) contact force of UAM in inspection task.

Figure 9. Snapshot of experiment in outdoor contact inspection.

4.3. Outdoor experiment

This experiment validates the effectiveness of the proposed control scheme for the UAM performing contact inspection with force tracking in an outdoor environment. In this experiment, UAM maintains the same contact force as in 4.2 to conduct contact inspection on the piers of outdoor bridges. All the parameters and initial status are the same as the previous experiment. The outdoor experiment scene is shown in Fig. 10. According to Fig. 9, considering the influence of outdoor wind, the fluctuation of the UAM in the whole inspection process is greater than that in the room. It can be seen from Fig. 10(e) that the fluctuation of contact force in the contact process is greater than that in the room, but it is still controlled at around 2.5 N.

Figure 10. Experiment result of UAM contact inspection with a desired force of $2.5\;\text{N}$ , the image feature markers were chosen as circular shape on a wooden board (25 $\times$ 55 cm). (a) Position trajectory of UAM platform in inspection task. (b) Angular velocity of UAM platform in inspection task. (c) Linear velocity of UAM platform in inspection task. (d) Attitude tracking error of geometric controller, (e) contact force of UAM in inspection task.

5. Conclusions

To address the stability and operability issues based on contact inspection, an autonomous control law is proposed in this paper, which can autonomously perform contact inspection tasks. We select the circular mark as the image feature for vision guidance, design the IBVS controller as the inner loop based on the traditional impedance model, and use the geometric control as the outer loop to maintain the stability of the contact force. To demonstrate this capability, we conduct an indoor flight experiment. In the experiment, UAM uses camera feedback to the onboard computer for processing and sends it to FCU for positioning and autonomous motion control. The attitude decoupling between the aerial manipulator and the UAV overcomes the instability characteristics of the traditional quadrotor underactuated. The contact force is kept within a certain range during the contact detection process.

In future work, we will focus on the feasibility of aerial contact inspection in practical application, and the main concerns are as follows. First, it is necessary to constrain the field of view. Circular image features are easy to lose when considering the wind speed and no field of view constraints. Second, studying more stable image feature extraction algorithms is necessary to ensure reliable application in actual scenes.