A novel approach to control four multi-rotor drones in cooperative paired control using relative Jacobian

Abstract

This work presents a new formulation to holistically control four cooperative multi-rotor drones controlled in two pairs. This approach uses a modular relative Jacobian with components consisting of the Jacobians of each individual drone. This type of controller relies mainly on the relative motion between the drones, consequently releasing unnecessary constraints inherent to the control of drones in absolute motion. We present the derivations of all the necessary equations of the modular relative Jacobian to control the four multi-rotor drones. We also present the derivations of the Jacobian for each drone. We implement our proposed method in the Gazebo RotorS simulation using four hexa-rotor drones modeled from Ascending Technologies called firefly drones. We present the simulation results and analyze them to show the effectiveness of our proposed approach.

1. Introduction

Drone control strategies significantly increased in the recent decade coupled with studies in cooperative drone control [1–3], with examples shown in refs. [4–7]. Cooperative control approaches are influenced by the application for which the system is to be used. Some of these approaches include aerial transportation [6], surveying and mapping [8], search and rescue [9, 10], reconnaissance [11], and many others. Many researchers are working on different classes of drone control approaches, for example, formation control, behavior-based approach, leader-follower, virtual leader, etc. [12]. In most of these methods, some excel and some have drawbacks. For example, studies show that the virtual leader and formation control classified methods are much reliable in modern war whereas, on the other hand, have problems when it comes to making full use of the neighbors’ communication information for other aspects of control [13].

This paper addresses the problem faced with the usual approach in controlling multi-rotor drones to maneuver in 3D space using tilting heuristics by running the rotors at different speeds as seen in Fig. 1 [14–17].

Figure 1. The usual way of controlling drone tilting heuristics during a right roll maneuver [16].

The use of relative Jacobian, however, increases the degrees of freedom in the null space. Another problem addressed is that most approaches to drone cooperative control use leader-follower and master-slave principles where the drones will rely on the leader drone. This approach is not reliable because when the master or leader drone fails, all the follower drones will also fail. Manually controlling the drone by radio controllers may not be advantageous because there is a chance of human errors in controlling the drones. With the Jacobian controller, the drones become autonomous. This work intends to derive and use new formulations of drone control via relative Jacobian to control four multi-rotor drones. In addition, we also derive and use new formulation in Jacobian-based control of individual drones. This type of controller allows centralized and decentralized control, including obstacle avoidance in the null space [18]. This work claims the following contribution: (1) a holistic control of four drones as a single robot, (2) paired control of drones similar to dual-arm controllers, (3) Jacobian-based control for each drone, and (4) modular relative Jacobian of four drones with components consisting of individual Jacobian.

The use of Jacobian to control a drone uses the same principle as the use of Jacobian to control a mobile base robot. The only difference is the additional degrees of freedom provided by the z-axis. In the use of Jacobian-based control for multi-drone systems (MDSs), the same principle is used when applied to a multi-arm manipulator system. The main reason for the identical principle is that the Jacobian is derived based on the relative movement of one end-effector with respect to the other. Thus, it is called a relative Jacobian. In the case of multi-arm manipulator systems, the relative movement between end-effectors releases unnecessary constraints in the null space and transforms them into controllable degrees of freedom. This affords more dexterity in the multi-arm manipulator system. The relative movement between end-effectors did not require a physical connection between them. In the same way, we use the relative Jacobian to control the relative movement between drones, where each drone contains both the base and end-effector of the “manipulator.” As in the case of a multi-arm manipulator system, the relative movement between drones does not require a physical connection between them. In addition, unnecessary constraints in the null space now become useful, controllable degrees-of freedom for the MDS.

2. Initial implementation of relative Jacobian controller

The Jacobian matrix brought up the solution to the inverse kinematics problem of robotics, this is the problem robots have achieving coordinated motion relative to a fixed reference frame [19]. The work in ref. [20] showed that the Jacobian matrix computes a transformation between the joint velocities and the end-effector velocities. A lot of research has been done on Jacobian control, especially with robot manipulators. Several studies have proved the relative Jacobian controller to be very satisfactory in robot manipulators, and several studies have been published; these include [21–23]. With these studies on the use of relative Jacobian on robot manipulators, we assume the derivation of relative Jacobian can be expressed for the desired number of manipulators or robots intended to be controlled. For example, the study in ref. [21] first introduced a modular relative Jacobian for a dual-arm robot in full space consisting of two manipulators $A$ and $B$ each with their absolute Jacobians $J_A=[J_{pA};\,J_{oA}]$ and $J_B=[J_{pB};\,J_{oB}]$ expressed in terms of position and orientation velocity components $J_{pA}, J_{oA}$ for manipulator $A$ and $J_{pB}, J_{oB}$ for manipulator $B$ .

This can be extended to more manipulators, like three (3) as in ref. [23] or four (4) in ref. [24]. Therefore, for the example of two manipulators, the modular relative Jacobian:

(1) \begin{equation} J_R= \begin{bmatrix} -^2\Psi _3 \: ^2\Omega _1 \: J_A & ^2\Omega _{4} \: J_B \end{bmatrix} \end{equation}

where

(2) \begin{equation} ^i\Psi _j= \begin{bmatrix} I & \quad -S\!\left (^ip_j\right ) \\ 0 & \quad I \end{bmatrix} \quad \mathrm{and}\quad ^i\Omega _j= \begin{bmatrix} ^iR_j & 0 \\ 0 & ^iR_j \end{bmatrix} \end{equation}

the symbol $^i\Psi _j$ is the wrench transformation matrix, $^i\Omega _j$ is the full space rotation matrix, $^ip_j$ is a position vector, $^iR_j$ is an orientation matrix, $S(^ip_j)$ is a skew-symmetric matrix, such that frame $j$ is expressed with respect to frame $i$ . The symbol $I$ is an identity matrix.

As depicted in (1), we can see that for one to derive the relative Jacobian $J_R$ , one only needs the wrench transformation matrix $^i\Psi _j$ , the rotation matrix $^i\Omega _j$ , and the Jacobians of the stand-alone manipulators $J_A$ and $J_B$ to form the relative Jacobian $J_R$ . So to extend this to four robots $A,\, B,\, C,$ and $D$ , for example, each robot’s absolute Jacobian $J_A$ , $J_B$ , $J_C$ , and $J_D$ are all incorporated to form the relative Jacobian of the four robots like in ref. [24].

The following paragraphs show the previous work of relative Jacobian implementation on other robots. This is the foundation of this paper’s implementation of cooperative drones.

A more compact expression of relative Jacobian based on individual manipulator Jacobians

In ref. [21], the first derivation of a more compact expression of relative Jacobian for parallel (dual-arm) manipulators was presented. Here, relative Jacobian was expressed in terms of the individual manipulator Jacobians. The aim is to control the combined manipulators as a single manipulator.

Modular relative Jacobian control for bipedal robot

A modular relative Jacobian method that was used for dual arms in ref. [21] was used for generating walking motion for bipedal robots [22]. The two legs that are being experimented with are equal to the two arms of the dual arm, whereas the feet here are the same as the two end-effectors of the dual arm. Each leg has three degrees of freedom (3-DOFs), so combined they form a 6-DOFs bipedal robot. The 3-DOFs relative position of the bipedal robot’s feet are controlled as the relative end-effectors of the relative Jacobian, and the other remaining 3-DOFs are used to control the posture of the robot.

Modular relative Jacobian for combined 3-arm parallel manipulators

The work in ref. [23] presented a formulation of a modular relative Jacobian for controlling combined manipulators as an individual manipulator having one end-effector. This relative Jacobian was designed for 3-arm parallel manipulators; it is expressed relative to the reference frames at the end-effector of the manipulators. The existing information about each stand-alone manipulator is used in the derivation of the relative Jacobian for the combined system. The motive behind the study of parallel robots includes several applications such as modular micro parallel robots [25], modular control architecture [26], modular designs of parallel robots, etc.

Bio-inspired holistic control through modular relative Jacobian for combined four-arm robots

This work was inspired by other living organisms having a different number of limbs which come in pairs, for example, mammals having four, insects with six limbs, arachnids having eight, etc [24]. This was an attempt to copy the biological method living organisms use in controlling paired legs. In this work, however they used dual arms in place of the paired legs. The experiment was to use these dual arms in opposite and adjacent pairs to achieve holistic control of a large four-legged animal. A modular relative Jacobian was used to control each dual arm as an individual manipulator with a single end-effector and is expressed in terms of the Jacobians of each of the stand-alone manipulators. In this work, the two opposite pairs of legs are treated as single end-effector dual arms, and then, these two dual arms are combined to form a single end-effector four-arm robot. The four-arm controller uses the same principle as a single end-effector controller of a dual arm and thus results in a single end-effector controller of a four-arm. A modular relative Jacobian of the four arms is derived.

3. Relative Jacobian cooperative controller implementation on multi-rotor drones

The intention of this paper, however, is to implement this controller on multi-rotor drones and test its efficiency as in cooperative robot manipulators. This work intends to apply the same concept of the relative Jacobian cooperative controller on multi-rotor drones in the same way it is implemented on cooperative manipulators.

The usual way multi-rotor drones maneuver is by using a tilt heuristic by running the rotors at different speeds [14, 15, 27]. This approach has unnecessary constraints, resulting in limited accuracy because there is less control over the movement of the drone. The tilt heuristics are used to compensate for the lack of actuation to change the xy-orientation (rotation about a horizontal axis), especially for quadrotors because they are under-actuated with respect to the full 6-DOF space. This is the case of a coupled movement, such that to change the xy-orientation, the position along the axis perpendicular to the axis of rotation also needs to change. However, this heuristic is not necessary for hexarotors if the propellers are properly configured because there are now six motors to move in the full 6-DOF space. If the propellers are configured properly, then each axis of movement can be independently controlled.

However, using Jacobian-based control the propellers have twist angles of 30 degrees about the $y$ -axis so that the drones move in a quintic path from their initial positions to the desired final positions, without tilting heuristics. The desired positions assigned $x_d$ , the desired orientation assigned $x_{do}$ , desired velocity $\dot{x}_d$ , and acceleration $\ddot{x}_d$ . Time $t$ is therefore input against all the aforementioned parameters which will result in $x_d(t) = [x_{dp}(t), x_{do}(t)], \dot{x}_d(t) = [\dot{x}_{dp}(t), \dot{x}_{do}(t)],$ and $\ddot{x}_d(t) = [\ddot{x}_{dp}$ $(t), \ddot{x}_{do}(t)]$ , respectively. Thus, the drone’s desired positions and orientations with their corresponding velocities and accelerations. On the assumption that each drone’s initial position or start, time, $t = 0$ , when the drone reaches the final desired position, the time will be $t = t_f$ . Therefore, we can consider the desired translational motion without any orientation as $\dot{x}_{dp}(0) = \ddot{x}_{dp}(0) = 0$ and $\dot{x}_{dp}(t_f) = \ddot{x}_{dp}(t_f)= 0$ .

The desired orientation $R_d$ is determined to compute the rotational motions for each rotor, also $^ip_i$ which is the drone’s arm length with respect to frame $i$ , plus the desired acceleration, $\alpha _d$ , and the rotational velocity, $\omega _d$ . In our quintic computations, these values are all initialized to zero, and the path planning of position and orientation use parameterized quintic functions. With regard to orientation, our quintic path is parameterized by $\theta _d$ , $\dot{\theta _d}$ , and $\ddot{\theta _d}$ , and the desired initial and final values for the quintic computations will be: $\theta _d(0)=\ddot{\theta }_d(0)=0,\dot{\theta }(t_f)=\ddot{\theta }_d(t_f)=0, \theta _d(0)=\theta _{\text{init}}$ , and $\dot{\theta }_d(t_f)=\theta _{\text{fin}}$ .

These desired orientations with respect to xyz-axis are used to compute the desired orientation as shown in (3) below:

(3) \begin{equation} R_d = R_{z,\phi _d}\,R_{y,\theta _d}\,R_{x,\psi _d} \end{equation}

Each individual drone’s rotor configurations are configured in the following manner with respect to its body, $^0R_i$ :

\begin{align*}{}^0R_1 &= R_{z,\phi }\!\left (\phi = 30^{\circ }\right )\, R_{y,\theta }\!\left (\theta =30^{\circ }\right ),\\{}^0R_2 &= R_{z,\phi }\!\left (\phi = 90^{\circ }\right )\, R_{y,\theta }\!\left (\theta = -30^{\circ }\right ),\\{}^0R_3 &= R_{z,\phi }\!\left (\phi = 150^{\circ }\right )\, R_{y,\theta }\!\left (\theta = 30^{\circ }\right ), \\{}^0R_{4} &= R_{z,\phi }\!\left (\phi = -150^{\circ }\right )\, R_{y,\theta }\!\left (\theta = -30^{\circ }\right ),\\{}^0R_5 &= R_{z,\phi }\!\left (\phi = -90^{\circ }\right )\, R_{y,\theta }\!\left (\theta = 30^{\circ }\right ), \text{ and}\\{}^0R_6 &= R_{z,\phi }\!\left (\phi = -30^{\circ }\right )\, R_{y,\theta }\!\left (\theta = -30^{\circ }\right ()\\ \end{align*}

The mass of the drone, $m=1.5\,\text{kg},{}^ip_i=[0.215,0,0.037]^T$ , matrix of mass inertia, $I_m=[0.0347563,0,0;\,0,0,0.0977]\,\text{kg}$ / $\text{m}^2$ , the moment proportionality, and rotor force moments constants, $a_f=[0,0,0.0045,0,0,0.0001]^T$ , these are all the same for all the four drones. The values of $a_f$ are derived empirically. With this, when a drone is moving from its start location towards its goal, there is minimal change in the orientation of the drone. This is because each axis is independently controlled, so when the drone changes position, this movement does not affect its orientation. This proves the fact that for a drone to move with fewer restrictions, all we need is the right combination of drone configurations.

This controller will be derived in two cases in the following subsections, 3.1 presents an individual or absolute Jacobian control of multi-rotor drones, and 3.2 presents the implementation of relative Jacobian cooperative control to control four multi-rotor drones in pairs.

3.1. Case 1: Derivation of individual Jacobian control of drones

As the first step to relative Jacobian derivation, we implemented absolute Jacobian for four individual drones. Figure 2 shows the graphical abstract of this implementation taking into consideration the kinematics and dynamics associated with drone control. This figure also shows the coordinate frames used in deriving the dynamic and kinematic models of individual drones. Each drone is attached a reference frame, frame $\{1\}$ is attached to drone 1, frame $\{2\}$ attached to drone 2, frame $\{3\}$ attached to drone 3, and frame $\{4\}$ attached to drone 4, all drones move with respect to frame 0 which is the world frame.

Figure 2. Demonstration of absolute Jacobian control of individual drones, each drone referencing the world frame.

For this implementation, the initial positions $[x,y,z]$ of the four drones are as follows, drone 1 $[0, 0, 0]$ , drone 2 $[0,2,0]$ , drone 3 $[1,0,0]$ , and drone 4 $[1,2,0]$ . And the desired final positions are drone 1 $[0, 3, 1]$ , drone 2 $[{-}2,6,1.5]$ , drone 3 $[2,-4,2]$ , and drone 4 $[0.5,5,2.2]$ . The Jacobian controller calculates the drone’s rotor speeds automatically. The rotor velocities and Jacobians for each individual drone $i$ are expressed with respect to the frame allocations in Fig. 2. The Jacobian for each drone $i$ , $J_i$ , is expressed as

(4) \begin{equation} J_i ={^w\Omega _0}\begin{bmatrix}^0\Omega _1\,^0\Omega _2\,^0\Omega _3\,^0\Omega _{4}\,^0\Omega _5\,^0\Omega _6\end{bmatrix}\,a_f \end{equation}

where $a_f$ is a scalar factor, $[^0\Omega _1 \ldots ^0\Omega _6]$ are the rotation matrices of each propeller frame $\{i\}$ with respect to the reference frame $\{0\}$ , and the superscript $\{w\}$ means the world frame, such that the rotation matrix is expressed as

(5) \begin{equation} ^i\Omega _j = \begin{bmatrix}^i\!R_j & \quad 0\\ 0 & \quad ^i\!R_j\end{bmatrix}. \end{equation}

Because this is an absolute controller, reference frame $\{0\}$ corresponds to the world frame $\{w\}$ . We define the velocity command for the $i$ th drone to be

(6) \begin{equation} V_i = J^T_i\, ^w\Omega _0 \, I_m \, u_i + J^T_i \, ^w\!\Psi _0 \, ^w\Omega _0 \, I_m \, \dot x+ J^T_i \, mg \end{equation}

where the terms are defined as

(7) \begin{equation} \,^w\!\Psi _0 = \begin{bmatrix}S(^w\!\omega _{\,0}) & \quad 0\\[2pt] 0 & \quad S(^w\!\omega _{\,0})\end{bmatrix} \end{equation}

(8) \begin{equation} u_i=\ddot x_d + k_v( \dot x_d - \, ^0\!\dot{x_i}) + k_p(x_d - \, ^0\!x_i). \end{equation}

The symbol $^w\omega _0$ is the angular velocity of the reference frame $\{0\}$ with respect to the world frame $\{w\}$ , $I_m$ is the mass inertia matrix of each individual drone, $u_i$ is the controller of drone $i$ , $^0x_i$ is the actual position of drone $i$ , $^0\dot{x}_i$ is the actual velocity of drone $i$ , and ${x}_d, \dot{x}_d, \ddot{x}_d$ are the desired position, velocity, and acceleration for both drones. The symbol $V_i$ is the rotor velocity for drone $i$ . The values of $k_v$ and $k_p$ are tuned empirically.

3.2. Case 2: Derivation of relative Jacobian for 4 drones

This section will show the derivation of relative Jacobian for four drones as pairs (2) and quads (4); this implementation is similar to ref. [21] on dual arms. In the first case, the four drones were controlled individually; now we will show the derivation of relative Jacobian motion control for the four drones as different pairs and then further expand it to controlling all four as one. For pairs of drones 1 and 2, this is achieved by controlling drone 2 which is an end-effector drone with respect to drone 1 which is the base drone and for the second pair drone 4 is the end-effector drone and is controlled with respect to drone 3. Drone 1 and drone 3 use the world frame as their reference. The frame assignments in Fig. 3 are used in the derivation of this combined relative Jacobian-based control for these drone pairs. The drones use odometry sensor data from motion sensors to estimate the change in position over time. The position $[x,y,z]$ is estimated relative to each drone’s starting location. This feedback position data of the base drones (1 and 3) are used to control the end-effector drones (2 and 4), and the results are compared with the desired position. This approach however does not use the leader-follower or master-slave theorems.

Figure 3. Relative Jacobian-based control demonstration with frame assignments.

Now we present the derivation of relative Jacobian cooperative control for pair $1$ (drone $1$ and drone $2$ ) and pair $2$ (drone $3$ and $4$ ). The position of the end-effector drones with respect to the base drones can be expressed as

(9) \begin{equation} ^1\!{p}_2 ={^1\!R_0}\!\left (^0\!p_2 - \,^0\!p_1\right ) \quad \mathrm{and}\quad ^3\!{p}_{4} ={^3\!R_0}\!\left (^0\!p_{4} - \,^0\!p_3\right ) \end{equation}

and the velocities of the end-effector drones with respect to the base drones can be expressed below by getting the time derivative of the expression above,

(10) \begin{equation} \begin{split} ^1\!{\dot p}_2 = \left [{-}{^1\!R_0} \, S\!\left(^0\!p_1\right) +{^1\!R_0} \, S\!\left(^0\!p_2\right)\right ] \, ^0\!\omega _1 -{^1\!R_0} \, ^0\!\dot p_1 +{^1\!R_0} \, ^0\!\dot p_2 &\\ ^3\!{\dot p}_{4} = \left [{-}{^3\!R_0} \, S\!\left(^0\!p_3\right) +{^3\!R_0} \, S\!\left(^0\!p_{4}\right)\right ] \, ^0\!\omega _3 -{^3\!R_0} \, ^0\!\dot p_3 +{^3\!R_0} \, ^0\!\dot p_{4} \end{split} \end{equation}

where $S$ is a skew-symmetric matrix. Because this controller takes a paired drone control, the reference frame $\{0\}$ corresponds to the world frame $\{w\}$ . The symbol $^1\!R_0$ is the rotation of the world frame with respect to drone 1, and $^3\!R_0$ is the rotation of the world frame with respect to drone 3, which are the base drones, $^0\omega _1$ and $^0\omega _3$ are the angular velocities of the base drones with respect to the world frame, and $^0p_1$ , $^0p_3$ and $^0p_2$ , $^0p_{4}$ are the odometry positions of the base drones and the end-effector drones, respectively. The end-effector drone velocities with respect to the world frame are $^0\!\dot p_2$ and $^0\!\dot p_{4}$ . The relative angular velocities for the first and second pairs are

(11) \begin{equation} \begin{split} ^1\!\omega _2 = -^1\!R_0 \, ^0\!\omega _1 +{^1\!R_0} \, ^0\!\omega _2 = \, ^1\!R_0 \, \begin{bmatrix}-I&I \end{bmatrix} \begin{bmatrix} ^0\!\omega _1\\ ^0\!\omega _2 \end{bmatrix}, &\\ ^3\!\omega _{4} = -{^3\!R_0} \, ^0\!\omega _3 +{^3\!R_0} \, ^0\!\omega _{4} ={^3\!R_0} \, \begin{bmatrix}-I&I \end{bmatrix} \begin{bmatrix} ^0\!\omega _3\\^0\!\omega _{4} \end{bmatrix} \end{split} \end{equation}

having $^0\omega _{_2}$ and $^0\omega _{_{4}}$ as the end-effector drone angular velocities with respect to the world frame, and $I$ is the identity matrix. The relative Jacobian for controlling the two pairs are

(12) \begin{equation} \begin{split}{}^1\!J_2=\begin{bmatrix} -{}^1\Omega _{_0}{}^1\Psi _2 \, \!J_1 &{}^1\Omega _0 \, \!J_2 \end{bmatrix}, &\\{}^3\!J_{4}=\begin{bmatrix} -{}^3\Omega _{_0}{}^3\Psi _{4} \, \!J_3 &{}^3\Omega _0 \, \!J_{4} \end{bmatrix} \end{split} \end{equation}

where ${^1\!J}_2$ and ${^3\!J}_{4}$ are the relative Jacobians for both relative motion controllers. The symbols ${}^1\Omega _0$ and ${}^3\Omega _0$ are the 6-DOF rotation of the world frame with respect to the base drone, $^1\Psi _2$ and $^3\Psi _{4}$ are the wrench transformation matrices of the end-effector frames with respect to the base frames, and $J_1, J_2, J_3, J_{4}$ are the absolute Jacobians of each drone.

To implement a full relative Jacobian-based cooperative controller, we need to use a torque controller instead of a velocity controller because the torque controller is more numerically stable when dynamics parameters are modeled. In our case, we modeled inertia and gravity parameters.

The relative position and relative velocity of frame $\{2\}$ with respect to frame $\{1\}$ , together with the relative Jacobian for the same frames, are shown below. This results in the rotor velocities of drone $1$ and drone $2$ . In the same way, the velocities of drone $3$ and drone $4$ are derived from the relative positions of frame $\{3\}$ and frame $\{4\}$ and are shown below

(13) \begin{equation} \begin{bmatrix} \dot \theta _1 \\ \dot \theta _2\ \end{bmatrix} ={^1\!J}_2^+ \begin{bmatrix} ^1{\!\dot p}_2 \\ ^1\!\omega _2\ \end{bmatrix} \quad \mathrm{and}\quad \begin{bmatrix} \dot \theta _3 \\ \dot \theta _{4}\ \end{bmatrix} ={^3\!J}_{4}^+ \begin{bmatrix} ^3{\!\dot p}_{4} \\ ^3\!\omega _{4}\ \end{bmatrix}. \end{equation}

The combined joint space inertia for both drone $1$ and drone $2$ , $D_{12}$ , and the combined joint space inertia for both drone $3$ and drone $4$ , $D_{34}$ , are shown below

(14) \begin{equation} \begin{split} D_{12} = &\begin{bmatrix} J^T_1 \, ^w\Omega _{1} \, I_m \, \!J_1 & \quad 0\\[3pt]0 & \quad J^T_2 \, ^w\Omega _{2} \, I_m \, \!J_2 \end{bmatrix} \\[4pt] D_{34} = &\begin{bmatrix} J^T_3 \, ^w\Omega _{3} \, I_m \, \!J_3 & \quad 0\\[3pt]0 & \quad J^T_{4} \, ^w\Omega _{4}\, I_m \, \!J_{4} \end{bmatrix} \end{split} \end{equation}

where ${}^w\Omega _1,{}^w\Omega _2,{}^w\Omega _3,{}^w\Omega _{4}$ are the 6-DOF rotation matrices of each drone with respect to the world frame $\{w\}$ , and $I_m$ is the inertia for each drone which is the same for all because the drones are identical.

We use relative task space inertias shown below,

(15) \begin{equation} \Lambda _{R12} ={^1\!J}_2 \, D^+_{12} \, ^1\!J^T_{2} \quad \mathrm{and}\quad \Lambda _{R34} ={^3\!J}_{4} \, D^+_{34} \, ^3\!J^T_{4} \end{equation}

and use dynamically consistent inverse [28] to compute the inverse of each relative Jacobian

(16) \begin{equation} \begin{split}{^1\!J}_2^+ = D^+_{12} \,{^1\!J}_2^T \,{\Lambda ^+_{R12}} \quad \mathrm{and}\quad{^3\!J}_{4}^+ = D^+_{34} \,{^3\!J}_{4}^T \,{\Lambda ^+_{R34}}. \end{split} \end{equation}

The null-space controller is shown below

(17) \begin{equation} {}^1Y_{R2} = I -{^1\!J}_2^+ \,{^1\!J}_2 \quad \mathrm{and}\quad{}^3Y_{R4} = I -{^3\!J}_{4}^+ \,{^3\!J}_{4} \end{equation}

where ${}^1Y_{R2}$ and ${}^3Y_{R4}$ are the null-space terms and $I$ is an identity matrix. The relative task space controllers are shown below

(18) \begin{equation} \begin{split}{}^1u_{R2} ={}^1\ddot x_{d2} + k_v\!\left ({}^1\dot x_{d2} - \, ^1\!\dot{x}_{_2}\right ) + k_p\!\left ({}^1x_{d2} - \,{^1\!x_{_2}}\right ) &\\{}^3u_{R4} ={}^3\ddot x_{d4} + k_v\!\left ({}^3\dot x_{d4} - \, ^3\!\dot{x}_{_{4}}\right ) + k_p\!\left ({}^3x_{d4} - \,{^3\!x_{_{4}}}\right ) \end{split} \end{equation}

where $^1x_{_2}$ , $^3x_{_{4}}$ are the relative positions and $^1\dot{x}_2$ , $^3\dot{x}_{4}$ are the relative velocities of the end-effector drone with respect to the base drone. The symbols ${}^1x_{d2},{}^1\dot x_{d2},{}^1\ddot x_{d2}$ are the desired position, velocity, and acceleration of drone $2$ with respect to drone $1$ , while ${}^3x_{d4},{}^3\dot x_{d4},{}^3\ddot x_{d4}$ are the desired position, velocity, and acceleration of drone $4$ with respect to drone $3$ . The translational parts of ${}^1x_2,{}^1\dot x_2$ are ${}^1p_2,{}^1\dot p_2$ , and the rotational parts are ${}^1R_2,{}^1 \omega _2$ , respectively. In same way, the translational parts of ${}^3x_{4},{}^3\dot x_{4}$ are ${}^3p_{4},{}^3\dot p_{4}$ , and the rotational parts are ${}^3R_{4},{}^3 \omega _{4}$ , respectively.

The velocity commands sent to each drone rotor are

(19) \begin{equation} \begin{split} \dot{q}_{12}= &\,{}^1J_2^+ \, \Lambda _{R12} \,{}^1u_{R2} + \,{}^1Y_{R2} \, \left (\!J_1^+ \,{}^w\Omega _{1} \, I_m \, u_1\right ) \\ \dot{q}_{34}= &\,{}^3J_{4}^+ \, \Lambda _{R34} \,{}^3u_{R4} + \,{}^3Y_{R4} \,\left (\!J_3^+ \,{}^w\Omega _{3} \, I_m \, u_3\right ) \end{split} \end{equation}

where $[\dot \theta _1, \dot \theta _2] = \dot q_{12}$ and $[\dot \theta _3, \dot \theta _{4}] = \dot q_{34}$ . Now we derive the relative Jacobian for motion control of four multi-rotor drones. The idea in this derivation is to combine the dual drone components the same way [24] combined manipulator components as dual arms. An effective individual drone relative Jacobian is applied for each dual-drone pair, this means that now we will have the four drone controller consisting of two drones expressed at the relative reference frames. These two drones in the relative reference frames can now then be combined again and be like a dual-arm robot with an individual manipulator controller. This will put the four drones in a holistic single-drone control through dual-drone pairing. With this implementation, the drones can be holistically controlled two at a time. The relative Jacobian gives us the ability to control different combinations simultaneously. Some of the possible combinations are shown in Eqs. (20) to (24). For dual drones $1$ and $2$ the relative Jacobian control equation is

(20) \begin{equation} {}_{4}^1J_2=\begin{bmatrix} -{}^1\Omega _0{}^1\Psi _2 J_1 & \,{}^1\Omega _0 J_2 &\, \mathbf{0} &\, \mathbf{0} \end{bmatrix}, \end{equation}

where $^i\Psi _j$ and $^i\Omega _j$ are as explained in (2). The relative Jacobian for dual drones $3$ and $4$ is

(21) \begin{equation} {}_{4}^3J_{4}=\begin{bmatrix} \mathbf{0} &\, \mathbf{0} &\, -{}^3\Omega _0{}^3\Psi _{4} J_3 & \,{}^3\Omega _0 \, J_{4} \end{bmatrix}, \end{equation}

the relative Jacobian for dual drones $1$ and $3$ is

(22) \begin{equation} {}_{4}^1J_3=\begin{bmatrix} -{}^1\Omega _0{}^1\Psi _3 J_1 &\, \mathbf{0} &\,{}^1\Omega _0 J_3 &\, \mathbf{0} \end{bmatrix}, \end{equation}

and the relative Jacobian for dual drones $1$ and $4$ is

(23) \begin{equation} {}_{4}^1J_{4}=\begin{bmatrix} -{}^1\Omega _0{}^1\Psi _{4} J_1 &\, \mathbf{0} & \mathbf{0} &\,{}^1\Omega _0 J_{4} \end{bmatrix}. \end{equation}

The relative Jacobians, in this case, have four columns corresponding to the four drones, such that column $1$ corresponds to drone $1$ , column $2$ corresponds to drone $2$ , etc. A zero column indicates that its corresponding drone lies in the null space. The velocity controller is used for this derivation, and the velocity control equation is

(24) \begin{equation} \begin{split} \dot{q}_{1234} = \ {{}^1_{4}J_2^+} \ \Lambda _{_R12} \ {}^1u_{R2} + (I -{}^1_{4}J_2^+ \ {}^1_{4}J_2) \ {_{4}J_1^+} \ {}^w\Omega _{1} \ I_m \ u_{_{1}} \ + \dots &\\ \dots +{}^3_{4}J_{4}^+ \ \Lambda _{R34} \,{}^3u_{R4} + (I-{}^3_{4}J_{4}^+ \ {}^3_{4}J_{4}) \ {_{4}J_3^+} \ {}^w\Omega _{3} \ I_m \ u{_3} \end{split} \end{equation}

where $_{4}J_1 = \begin{bmatrix} J_1 & \mathbf{0} & \mathbf{0} & \mathbf{0} \end{bmatrix}$ and $_{4}J_3 = \begin{bmatrix} \mathbf{0} & \mathbf{0} & J_3 & \mathbf{0} \end{bmatrix}.$

4. Experimental setup

This is a simulation-based research project, we ran it on an open-source operating system software Linux Ubuntu 18.04. It runs from the desktop to the cloud and all your Internet-connected things. Below is the discussion of the developing tools that are required in setting up our environment.

4.1. Robot operating system

Robot operating system (ROS) is an open-source software designed to meet specific challenges robot developers encounter when developing large-scale service robots [29, 30]. ROS has low-cost sensors that bring unquestionably advantages to the robot developers, it gives access to a wide range of robotic applications for any user or developer. It has led to rigorous progressive advances in the development of robotic solutions and has thus become more widespread among the robotics society [31].

ROS consists of several important parts [32]:

1. 1. A set of drivers that reads data from sensors of the simulated robot and sends commands to motors and actuators.

2. 2. A collection of robotic algorithms that can be used to build maps of the world, navigate around the built world, represent and interpret sensor data, manipulate objects, plan motions, etc.

3. 3. Computational infrastructure that can move data around connecting different components of a complex robot system and allows the developer or researcher to incorporate their algorithms.

4. 4. A set of tools like Robot state visualization tools, and tools that make it easy to visualize the algorithms, debug errors, and record sensor data.

5. 5. Extensive set of resources like wiki which documents many aspects of ROS.

4.2. Gazebo simulator

Gazebo simulator is the product of the University of Southern California, which was developed in 2002. The sole purpose of Gazebo was to make a high-fidelity simulator that makes it possible to simulate robots in outdoor environments under various conditions same as in the real world. Gazebo simulator can test the high-performance physics open dynamics engine and the robot sensors for prototype development activities [33]. Gazebo works with ROS together making a powerful combination with the ability to simulate with credibility, flexibility, robustness, and capability of supporting multiple robots performing complex tasks in various distributed environments [34]. Gazebo has various examples of robotic simulations that are easy to find using the models supplied by Gazebo manufacturers. Gazebo simulates three-dimensional dynamics for mechanisms of single and multiple robots, for both inside and outside environments. Even though Gazebo was created to relieve robot developers from the realistic robot simulation in outdoor environments, they mostly use it for indoor simulations. This simulator creates realistic worlds for the robots, and it relies entirely on physics-based characteristics, reflecting the physics when the robot model is pulled, knocked, pushed, etc. It is also an open-source software.

4.3. RotorS simulation framework

RotorS simulation framework is a Gazebo-based micro aerial vehicle platform developed to reduce field testing times and to separate problems of testing, thus making it easier to debug [35]. It was developed by Autonomous Systems lab at ETH Zurich, Switzerland. The simulator has simulated sensors such as an IMU, a generic odometry sensor, and the VI-Sensor, which can be mounted on the simulated drone. The multi-rotor models provided by rotorS include AscTec Hummingbird, AscTec Pelican, and the AscTec Firefly; however, the simulator is not limited to these models only. This paper uses the firefly, which is a hexa-rotor drone. The package comes with example controllers that lay the foundation for researchers and developers to build their controllers from.

5. Implementation results

This section lays out the results of our implementation, and the experiment is divided into two cases in which the first case shows results of individually controlled drones and the second case showing the results of cooperative control of four drones in pairs of two. For both the experiments, the drones’ initial positions $[x,y,z]$ are as shown in Table I. These positions are as visualized in Fig. 4

Table I. Individual drones’ initial positions.

Figure 4. Drones’ initial start positions for our experiments.

5.1. Case 1: Individual Jacobian control results

Each drone is controlled by its absolute Jacobian $J_1, J_2, J_3,$ and $J_{4}$ , respectively. The drones do not communicate or depend on other drones rather they exist as stand-alone, any drone can be given any desired final position to fly to without taking into consideration the positions of other drones. Each drone’s rotor velocities are computed in (6), and each is controlled using the world frame, { $0$ }, as their reference individually. For this experiment, the desired final positions are as projected in Table II.

Table II. Individual drones’ desired final positions.

The results of this experiment show each drone moving in 3D space to its desired final position in a quintic path without tilting heuristics. Figures 5 and 6 show the demonstration of absolute Jacobian-controlled drones in 3D space, with Fig. 6 showing when they have reached respective desired positions. Graphs shown in Fig. 7 are the feedback from the drone’s odometry sensors showing the positions of each drone when moving to their respective desired final positions, and these results show the accuracy of this controller having only small discrepancies which may be the because of the world factors like wind.

Figure 5. Individual Jacobian-controlled drones maneuvering in 3D space to their own assigned desired final positions, the drone does not tilt heuristics during this maneuver.

5.2. Case 2: Results of cooperative control of four drones

The implementation of the relative Jacobian cooperative controller on four drones is inspired by the implementation of this controller on a four-arm robot [24]. The study aimed at developing a holistic controller for controlling the four-legged animal. The attempt was to interpret the biological method of controlling two paired legs in adjacent and opposite pairs, and they used robot arms to represent the four legs. In our experiment, we used firefly drones instead of robot arms. The purpose of this simulation is to show the overall coordination of the four drones controlled under one individual end-effector drone controller. It is the extension of the concept of individual control where only drone 1 and drone 3 use the world frame as their reference and hence calling them the base drones. And drone 2 uses drone 1 as its reference and drone 4 using drone 3 as its reference hence calling them end-effector drones. Even though this implementation was first inspired by the biological limbs of a four-legged animal, this simulation does not mimic the walking of a live animal but uses the principle of coordination between adjacent, opposite, and diagonal pairs. Like in the four-legged animal, there is relative motion among the drones also the concept of controlling the drones is inspired by the way the animal minds control the four legs holistically. The drones are all controlled as if only one drone was being controlled.

Figure 6. Individual Jacobian-controlled drones hoovering at their respective desired final positions.

Figure 7. Odometry position feedback of each drone with the desired positions.

For this Case 2, we control the drone positions by using the base drones drone 1 and drone 3. Table III shows the base drones’ set desired final positions used for this experiment. The relative positions of drone 2 and drone 4, the end-effector drones with respect to the base drones, have also been set. When the base drones move, the end-effector drones will follow accordingly, thus moving with respect to the respective base drone because the desired relative position is maintained; thus remains the same. This is as shown in Fig. 8.

Figure 8. Cooperative drones hovering at their desired final positions.

Figures 9 and 10 show the odometry position comparisons with the desired positions of both pairs. These graphs show the positions of these drones from the initial positions stated in Table I to the desired positions stated in Table III. Note that here the positions of firefly 2 and firefly 4 are the desired relative positions of the end-effector drones, we only change the desired final positions for the base drones only, that is, drone 1 and drone 2 (firefly 1 and firefly 3) and the end-effector drones will follow maintaining the stated desired relative positions.

Figure 9. Pair 1, drone 1 and drone 2 odometry position feedback with their desired positions.

Figure 10. Pair 2, drone 3 and drone 4 odometry position feedback with their desired positions.

Table III. Cooperative-controlled drone pairs desired final positions.

The general response on positional error of cooperative drones maneuvering from initial position to desired final position is shown in Fig. 11; it depicts the average performance of the relative Jacobian cooperative-controlled drones. The graph shows the error reduction as the drone is maneuvering to its desired final position until the error becomes zero. There is also minimal change in orientation of the drone. It demonstrates the fact that given the right combination of drone configurations, the drones are able to maneuver with less restrictions.

Figure 11. Performance of the base drone among the relative Jacobian cooperatively controlled drones.

6. Potential applications

Jacobian-based control and cooperative-controlled multi-rotor drones can move precisely because of no tilting heuristics and hence can suit many applications that require precise movements. With the use of good machine learning techniques, relative Jacobian-controlled team of drones can make good surveillance because the drones are stable; hence, the camera technology will give accurate results.

6.1. Unexploded ordnance disposal

During military training and operations, the use of antitank weapons like grenade launchers, for example, Rocket Propelled Grenade 7 (RPG 7), 84 mm Carl Gustav, etc., is highly used. In some most unfortunate situations, these weapons can have horrible misfired shells or unexploded ordnances. Ammunition is a very complex mechanism that consists of many dependent and interrelated functions; in the event of unexploded ordnance, the fuzing system is most probably the primary point of failure [36]. This however poses safety hazards to the environment [37], and therefore, the need for misfires and unexploded ordnances to be disposed. They can also explode during the disposal period and probably causing harm to humanity. So, multi-rotor drones with attached robot arms or sanction pads can be an alternative way of disposing of this ammunition to hinder the harm that can occur to humanity during disposal. Here now we will need a drone that will be mostly stable because disturbances to the unexploded ordnance may cause it to explode. Jacobian-controlled multi-rotor drones move in 3D space without tilting of heuristics this will be the best controller to use for the application of misfired shells or unexploded ordnances because of high stability during maneuvering.

6.2. Patrols and videography

As videography is one of the most common application of multi-rotor drones, there are a lot of applications of multi-rotor drones based on videography. This is because videography is relatively not expensive to conduct due to the low costs of viewing devices like cameras and low costs for operator training [38]. Sometimes, there is a need to record steady and more concentrated videos; this can be useful in filmmaking or patrols performed by the military or police. This will require a more stable drone, and if there is a need to capture a large area, relative Jacobian-based cooperative multi-rotor drones can be used. Cooperative drones controlled with this controller can be an effective supplement to the current patrolling business in road traffic patrolling with complex urban buildings and road conditions and a limited ground perspective. One of the problems that drones on patrol is having the patrolling cooperative drones visit and revisit the target area to cover the areas that may have been missed [39, 40]. Relative Jacobian-based cooperative control is the solution to this problem because the drones are more stable than other drone controllers and hence will not miss areas to be covered. The captured videos can be merged, and it will be as if only one large coverage camera was used. This shows the good outcomes of the relative Jacobian-based cooperative controller.

7. Conclusions and future work

This study showed implementation results of individual Jacobian control and relative Jacobian-based cooperative motion control of AscTec firefly hexa-rotor drones using RotorS Gazebo platform. A new formulation is presented that allows Jacobian-based control of individual drone and relative Jacobian-based control of a pair of drones in terms of relative position and orientation. Lastly, a new formulation is presented that allows relative Jacobian-based control of four drones, using paired relative position and orientation control. However, in these implementation relative orientation remained constant and only relative position control is changed. In future work, this controller will be implemented on real hardware. Work is in progress to build own DIY drones to implement this controller on. Another platform to implement this controller on is the DJI M200 v2 series and DJI Matrice 300 drones. Also as part of future work, both position and orientation control will be implemented on both individual and four cooperative drones.