Qualitative properties of solutions for system involving the fractional Laplacian

Abstract

In this paper, we consider the following non-linear system involving the fractional Laplacian0.1

\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u (x)= f(u,\,v), \\ (-\Delta)^{s} v (x)= g(u,\,v), \end{array} \right. \end{equation}

in two different types of domains, one is bounded, and the other is an infinite cylinder, where $0< s<1$. We employ the direct sliding method for fractional Laplacian, different from the conventional extension and moving planes methods, to derive the monotonicity of solutions for (0.1) in $x_n$ variable. Meanwhile, we develop a new iteration method for systems in the proofs. Hopefully, the iteration method can also be applied to solve other problems.

1. Introduction

The work concerning qualitative properties of solutions was extensively investigated by many researchers. Fruitful results have been obtained on the existence, non-existence, symmetry and regularity and so on. Berestycki and Nirenberg [3] obtained the monotonicity for the unbounded positive solutions of elliptic equations in the case of a ‘coercive’ Lipschitz graph. In [6], Berestycki et al. proved the monotonicity and uniqueness for elliptic equations in unbounded Lipschitz domains. Angenent [1] and Clément and Sweers [15] derived that a bounded positive solution of elliptic equations only depends on the $x_n$-variable in a upper half space. Chen et al. [12] worked on the symmetry and non-existence of positive solutions of equations with fractional Laplacian in different types of domains. For more related results, please see [7, 10, 11, 14, 18], and the references therein.

In this work, we investigate the following system involving the fractional Laplacian:

(1.1)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u (x)= f(u,\,v), \\ (-\Delta)^{s} v (x)= g(u,\,v) \end{array} \right. \end{equation}

in a bounded domain and an unbounded domain, respectively.

From an applicable point of view, the fractional Laplacian have caught researchers’ attention because of its non-locality and its applications in physical sciences. So far it has been utilized to model diverse physical phenomena, such as anomalous diffusion and quasi-geostrophic flows, turbulence and water waves, molecular dynamics and relativistic quantum mechanics of stars (see [5, 9, 16, 22] and the references therein). It also has various applications in probability and finance [2, 4]. In particular, the fractional Laplacian can be understood as the infinitesimal generator of a stable Lévy diffusion process [4].

The fractional Laplacian is a non-local pseudo-differential operator, taking the form

(1.2)\begin{equation} (-\Delta)^{s} u(x) = C_{n,\alpha} \, PV \int_{R^n} \frac{u(x)-u(z)}{|x-z|^{n+2s}} \,{\rm d}z,\end{equation}

where $s\in (0,\,1)$ and PV stands for the Cauchy principal value. This operator is well defined for $u\in C^{\infty }_0(R^n)$. In this space, it can also be defined equivalently in terms of the Fourier transform

\[ (-\Delta)^{s} u (x) = {\mathcal{F}}^{{-}1}(|\xi|^{2s} {\mathcal{F}}u(\xi))(x), \]

where $\mathcal {F}$ is the Fourier transform, and ${\mathcal {F}}^{-1}$ is the inverse Fourier transform. The fractional Laplacian can be extended to locally integrable functions with certain growth control—the weighted $L^1$-space:

\[ \mathcal{L}_{2s}=\{u: R^n\rightarrow R \mid \int_{R^n}\frac{|u(x)|}{1+|x|^{n+2s}} \, d x <\infty\} \ \mbox{(see [21])}. \]

For $u\in \mathcal {L}_{2s}$, we define $(-\Delta )^{s} u$ as a distribution:

\[ ((-\Delta)^{s}u)( \phi) = \int_{R^n} u(x) (-\Delta)^{s} \phi(x)\,{\rm d}x, \quad \forall\ \phi \in {C^{\infty}_c}(R^n). \]

To investigate the properties of solutions of equations involving the fractional Laplacian, the method of moving planes and extension method [8] have been powerful tools. In [25], the authors employed the method of moving planes in integral forms (see [10, 11, 13] and the references therein) to study the symmetry of solutions. However, this method needs to establish the equivalence between the differential equations and the integral equations via Green's functions, and it is a challenge work. Also, this method depends heavily on some special properties of the corresponding Green's functions. So far, there are few results about the Green's functions in general domains. Chen et al. [12] developed a direct method of moving planes for the fractional Laplacian. Based on the classical sliding method for Laplacian [3], Wu and Chen [23, 24] developed the direct sliding method for the scalar equations involving the fractional Laplacian. Actually, the direct sliding method does not depend on the Green's functions. In this paper, we employ the direct sliding method [23, 24] to derive the monotonicity of solutions of system (1.1) in a general bounded domain and an infinite cylinder.

The analogue problem to (1.1) for the fractional Laplacian has been investigated by many authors:

(1.3)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u_i (x)= f_i(u_1,\dots,u_m), \ x\in \Omega, \ & i=1,\dots,m, \\ u_i(x)=0, \ x\in {R}^n\setminus \Omega, \ & i=1,\dots,m. \end{array} \right. \end{equation}

In the case $\Omega$ be a unit ball or half space, Mou [19] proved the symmetry and monotonicity of positive solutions of (1.3) by the integral equation approach. When $i=1,\,2$, $f_1=u^p_2$, $f_2=u^q_1$ and $\Omega =R^n_+$, Quaas and Xia [20] obtained the non-existence of positive solutions of (1.3) by the method of moving planes with an improved Aleksandrov–Bakelman–Pucci type estimate for the fractional Laplacian.

In this paper, we consider the non-linear equations involving the fractional Laplacian in general domains. Due to the non-local nature of the operator, we need to set exterior conditions in domain $\Gamma$,

\[ u(x)=\varphi(x),\quad v(x)=\psi(x), \quad x\in \Gamma^c. \]

In order to ensure the monotonicity of solutions, one has to impose the necessary exterior conditions (P) on $\Gamma$: For any three points $x^1=(x',\,x^1_n)$, $x^2=(x',\,x^2_n)$ and $x^3=(x',\,x^3_n)$ with $x^1_n< x^2_n< x^3_n$, $x'\in {R}^{n-1}$, and $x^1$, $x^3\in \Gamma ^c$, $x^1$, $x^2$ and $x^3$ satisfy

\begin{align*} & \varphi(x^1)< u(x^2)<\varphi(x^3), \quad \mbox{for }x^2\in \Gamma,\\ & \varphi(x^1)\leq\varphi(x^2)\leq\varphi(x^3),\ \mbox{for }x^2\in \Gamma^c,\\ & \psi(x^1)< v(x^2)<\psi(x^3),\ \mbox{for }x^2\in \Gamma,\\ & \psi(x^1)\leq\psi(x^2)\leq\psi(x^3),\ \mbox{for }x^2\in \Gamma^c. \end{align*}

We first study (1.1) in bounded domain $G$, and establish monotonicity of solutions for (1.1) as following:

Theorem 1.1 Let $u$, $v\in \mathcal {L}_{2s}\cap C^{1,1}_{loc}(G)\cap C(\bar {G})$ and $(u,\,v)$ be a pair of solution for

(1.4)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u (x)= f(u,\,v), \ & x\in G, \\ (-\Delta)^{s} v (x)= g(u,\,v), \ & x\in G, \\ u(x)=\varphi(x),\ v(x)=\psi(x),\ & x\in G^c, \end{array} \right. \end{equation}

where $G$ is a convex bounded domain in $x_n$ direction, and $u$, $v$ satisfy the exterior condition $(\mathbf{P})$ on $G$. If $\frac {\partial f}{\partial v}>0$, $\frac {\partial g}{\partial u}>0$ for $x\in G$, and $\frac {\partial f}{\partial u}$, $\frac {\partial f}{\partial v}$, $\frac {\partial g}{\partial u}$, $\frac {\partial g}{\partial v}$ are bounded from above in $G$, then $u$ and $v$ are monotone increasing with respect to $x_n$-variable in $G$. More precisely, for any $\tau >0$, one has

\[ u(x',\tau+x_n)>u(x',x_n),\quad \forall\ (x',\tau+x_n), \,(x',x_n)\in G. \]

In the proof of theorem 1.1, employing the argument by contradiction at extreme points, we first derive the key tool—narrow region principle in bounded domains. Combining with the sliding method, the solutions of system (1.4) are proved to be monotonic in the bounded domain $G$. Next we briefly introduce the sliding method. For any positive real number $\tau$, by sliding downward $\tau$ units from the bounded domain $G$, we have

\[ G_{\tau}=\{x-\tau e_n\,|\,x\in G\}, \]

here $e_n=(0,\,\dots,\,0,\,1)$. Denote

\begin{align*} & u^{\tau}(x)=u(x',x_n+\tau),\quad v^{\tau}(x)=v(x',x_n+\tau),\\ & \tilde{U}^{\tau}(x)=u^{\tau}(x)-u(x),\quad \tilde{V}^{\tau}(x)=v^{\tau}(x)-v(x). \end{align*}

For $\tau$ sufficiently close to the width of $G$ in $x_n$ direction, it is easy to see that $G\cap G_{\tau }$ is a narrow region. Applying the narrow region principle in bounded domains yields that

(1.5)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq 0,\quad x\in G\cap G_{\tau}.\end{equation}

Note that (1.5) provides a starting position to slide the domain $G_{\tau }$. Then we slide $G_{\tau }$ back upward as long as inequality (1.5) holds to its limiting position. In fact, the domain should be slid to $\tau =0$. We conclude that the solutions of (1.4) are monotone increasing in $x_n$ variable.

Considering the unbounded domain $\Omega$, because of the unboundedness property for $\Omega$, the extremum points in $\Omega$ cannot be attained which makes it hard to apply the narrow region principle in unbounded domains directly. To overcome this difficulty, we first take a minimization sequence approaching to the infimum, and then make some perturbation about the sequence to attain the extremum points in some bounded domain. Combining with the iteration method, we can deduce the narrow region principle in unbounded domains. Then the monotonicity of solutions for (1.1) in an unbounded domain $\Omega$ is obtained with the aid of the direct sliding method.

Theorem 1.2 Let $u$, $v\in \mathcal {L}_{2s}\cap C^{1,1}_{loc}(R^n)$ be a pair of solution for

(1.6)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u (x)= f(u,\,v), \ & x\in \Omega, \\ (-\Delta)^{s} v (x)= g(u,\,v), \ & x\in \Omega, \\ u(x)=\varphi(x),\ v(x)=\psi(x),\ & x\in \Omega^c, \end{array} \right. \end{equation}

where $\Omega =\{x=(x',\,x_n)\in R^n\,|\,0< x_n< M\}$, $x'=(x_1,\,x_2,\,\dots,\,x_{n-1})$, and $M$ is a finite positive real number. $u,\,v$ satisfy the exterior condition $(\mathbf{P})$ on $\Omega$, and $u(x',\,\cdot )$ and $v(x',\,\cdot )$ are bounded with $x'\in R^{n-1}$. Suppose that $\frac {\partial f}{\partial v}>0$, $\frac {\partial g}{\partial u}>0$ for $x\in \Omega$, and $\frac {\partial f}{\partial u}$, $\frac {\partial f}{\partial v}$, $\frac {\partial g}{\partial u}$, $\frac {\partial g}{\partial v}$ are bounded from above in $\Omega$. Then $u(x)$ and $v(x)$ are monotone increasing in $x_n$-variable, that is, for any $\tau >0$,

\[ u(x',\tau+x_n)>u(x',x_n),\quad \forall (x',\tau+x_n), \,(x',x_n)\in \Omega. \]

Remark 1.3 Theorem 1.2 still holds if $\Omega$ is any domain bounded in the $x_n$-direction.

One of the interesting point about the monotonicity of solutions is that it helps to pave the way for deriving existence, non-existence and some Sobolev inequalities, as can be seen in [13, 14, 17] and the references therein.

This paper is organized as follows. In § 2, we establish some lemmas, such as the narrow region principle in bounded domains and so on. In § 3 and 4, combining the lemmas in § 2 with the sliding method, we derive the monotonicity of solutions for (1.1) on bounded domains and unbounded domains.

2. Key tools in the sliding method

The aim of this section is to show the key tools in the sliding method. More precisely, we investigate the narrow region principle in bounded domains and unbounded domains so that the sliding method can be initiated.

Lemma 2.1 Narrow region principle for system in bounded domains

Let $\tilde {U}$, $\tilde {V}\in \mathcal {L}_{2s}\cap C^{1,1}_{loc}(E)$ satisfy

(2.1)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} \tilde{U} (x)-b_1(x)\tilde{U}(x)-c_1(x)\tilde{V}(x)\geq 0, \ & x\in E, \\ (-\Delta)^{s} \tilde{V} (x)-b_2(x)\tilde{U}(x)-c_2(x)\tilde{V}(x)\geq 0, \ & x\in E, \\ \tilde{U}(x)\geq 0,\ \tilde{V}(x)\geq 0,\ & x\in E^c, \end{array} \right. \end{equation}

where $E$ is a bounded domain, $c_1(x)>0$, $b_2(x)>0$ in $E$, and $b_i$, $c_i$ are bounded from above in $E$, $i=1,\,2$. Then for $d$ sufficiently small, which is the width of $E$ in $x_n$ direction, one has

(2.2)\begin{equation} \tilde{U}(x)\geq 0,\quad \tilde{V}(x)\geq 0,\quad x\in E. \end{equation}

Proof. If (2.2) is not valid, then at least one of $\tilde {U}$ and $\tilde {V}$ is less than zero at some point. We may assume that there exists a point $x^0\in E$ such that

\[ \tilde{U}(x^0)=\min\limits_{R^n}\tilde{U}(x)<0. \]

By (2.1), for $d$ sufficiently small, we have

(2.3)\begin{align} 0& \leq (-\Delta)^{s} \tilde{U} (x^0)-b_1(x^0)\tilde{U}(x^0)-c_1(x^0)\tilde{V}(x^0) \nonumber\\ & \leq \frac{c\tilde{U}(x^0)}{d^{2s}}-b_1(x^0)\tilde{U}(x^0)-c_1(x^0)\tilde{V}(x^0) \nonumber\\ & \leq \frac{c\tilde{U}(x^0)}{d^{2s}}-c_1(x^0)\tilde{V}(x^0). \end{align}

This implies that

(2.4)\begin{equation} \tilde{V}(x^0)<0, \end{equation}

and

(2.5)\begin{equation} \tilde{V}(x^0)\leq \frac{c\tilde{U}(x^0)}{d^{2s}c_1(x^0)}. \end{equation}

It follows from (2.4) that there exists some point $x^1\in E$ such that

\[ \tilde{V}(x^1)=\min\limits_{R^n}\tilde{V}(x)<0. \]

Similar to (2.3), we derive that, for $d$ sufficiently small,

(2.6)\begin{align} 0& \leq (-\Delta)^{s} \tilde{V} (x^1)-b_2(x^1)\tilde{U}(x^1)-c_2(x^1)\tilde{V}(x^1) \nonumber\\ & \leq \frac{c'\tilde{V}(x^1)}{d^{2s}}-b_2(x^1)\tilde{U}(x^0). \end{align}

Moreover, we get

(2.7)\begin{equation} \tilde{U}(x^0)\leq \frac{c'\tilde{V}(x^1)}{d^{2s}b_2(x^1)}. \end{equation}

Combining (2.5) and (2.7) yields that

(2.8)\begin{equation} \tilde{V}(x^0)\leq \frac{c c'\tilde{V}(x^0)}{d^{4s}c_1(x^0)b_2(x^1)}. \end{equation}

Thus, one has

(2.9)\begin{equation} 1\geq\frac{c c'}{d^{4s}c_1(x^0)b_2(x^1)}. \end{equation}

(2.9) is impossible for sufficiently small $d$. Therefore, (2.2) holds.

Lemma 2.2 Narrow region principle for system in unbounded domains

Let $D_1=\{x=(x',\,x_n)\in R^n|0< x_n<2l\}$ be an unbounded narrow region with some bounded constant $l$, and $D_-=\{x=(x',\,x_n)\in R^n|x_n<0\}$. If ${U}$, ${V}\in \mathcal {L}_{2s}\cap C^{1,1}_{loc}(D_1)$ satisfy

(2.10)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} {U} (x)-\bar{b}_1(x){U}(x)-\bar{c}_1(x){V}(x)\geq 0, \ & x\in D_1, \\ (-\Delta)^{s}{V} (x)-\bar{b}_2(x){U}(x)-\bar{c}_2(x){V}(x)\geq 0, \ & x\in D_1, \\ {U}(x)\geq 0,\ {V}(x)\geq 0,\ & x\in D_-. \end{array} \right. \end{equation}

Suppose that $\bar {c}_1(x)>0$, $\bar {b}_2(x)>0$ in $D_1$, and $\bar {b}_i$, $\bar {c}_i$ are bounded from above in $D_1$, $i=1,\,2$. Then for $l$ sufficiently small, we get

(2.11)\begin{equation} {U}(x)\geq 0,\quad {V}(x)\geq 0,\quad x\in D_1. \end{equation}

Proof. The argument, by contradiction, is standard. Suppose (2.11) is false. Then at least one of ${U}(x)$ and ${V}(x)$ are less than zero at some points belonging to $D_1$. Without loss of generality, we may assume that there are some points such that the values of $U$ at these points are less than zero. Then there exists a sequence $\{x^k\}^{\infty }_{k=1}\subset D_1$ such that

(2.12)\begin{equation} U(x^k)\rightarrow A=\inf\limits_{R^n}U(x)<0, \end{equation}

with $|x^k_n|< l$, where $x^k_n$ is the $n$-th component of $x^k$.

Let

(2.13)\begin{equation} \eta(x)=\left\{\begin{array}{@{}ll} ae^{\dfrac{1}{|x|^2-l}}, \ & |x|< l, \\ 0, \ & |x|\geq l, \end{array} \right. \end{equation}

taking $a=e^{1/l}$ such that $\eta (0)=\max \limits _{R^n}\eta (x)=1$.

Set $\varphi _k(x)=\eta (x-x^k)$. Combining with (2.12), there exists a positive sequence $\{\epsilon ^k\}^{\infty }_{k=1}$ such that

(2.14)\begin{equation} U(x^k)-\epsilon^k\varphi_k(x^k)< A<0, \end{equation}

where $\epsilon ^k\rightarrow 0$ as $k\rightarrow \infty$.

Obviously, for $x\in R^n\setminus B_l(x^k)$, $U(x)\geq A$ and $\varphi _k(x)=0$. Then we have

(2.15)\begin{equation} U(x^k)-\epsilon^k\varphi_k(x^k)< U(x)-\epsilon^k\varphi_k(x),\quad \mbox{for }x\in R^n\setminus B_l(x^k), \end{equation}

here $B_l(x^k)=\{x\in R^n\,|\,|x-x^k|< l\}$.

Define $U_k(x)=U(x)-\epsilon ^k\varphi _k(x)$. It follows from (2.15) that there exists some point $\bar {x}^k\in B_l(x^k)$ such that

\[ U_k(\bar{x}^k)=\min\limits_{R^n}U_k(x)<0. \]

It is easy to see that

\[ U(\bar{x}^k)\leq U(x^k), \]

and

\[ U(\bar{x}^k)\rightarrow A,\quad \mbox{as }k\rightarrow\infty. \]

Applying the first inequality of (2.10) and the definition of the fractional Laplacian, we derive

(2.16)\begin{align} 0& \leq (-\Delta)^{s} {U} (\bar{x}^k)-\bar{b}_1(\bar{x}^k){U}(\bar{x}^k)-\bar{c}_1(\bar{x}^k){V}(\bar{x}^k) \nonumber\\ & = (-\Delta)^{s} {U}_k (\bar{x}^k)-\bar{b}_1(\bar{x}^k){U}_k(\bar{x}^k)-\bar{c}_1(\bar{x}^k){V}(\bar{x}^k) \nonumber\\ & \quad + \epsilon^k (-\Delta)^{s}\varphi_k(\bar{x}^k)-\epsilon^k\bar{b}_1(\bar{x}^k)\varphi_k(\bar{x}^k) \nonumber\\ & \leq (\frac{C}{l^{2s}}-\bar{b}_1(\bar{x}^k)) {U}_k(\bar{x}^k)-\bar{c}_1(\bar{x}^k){V}(\bar{x}^k)\nonumber\\ & \quad + \epsilon^k (-\Delta)^{s}\varphi_k(\bar{x}^k) -\epsilon^k \bar{b}_1(\bar{x}^k)\varphi_k(\bar{x}^k). \end{align}

Then for sufficiently small $l$ and sufficiently large $k$, one has

(2.17)\begin{equation} 0\leq \frac{C}{l^{2s}} {U}_k(\bar{x}^k)-\bar{c}_1(\bar{x}^k){V}(\bar{x}^k)+ o(\epsilon^k). \end{equation}

This implies that

(2.18)\begin{equation} {V}(\bar{x}^k)<0.\end{equation}

Based on (2.18), there exists a sequence $\{z^k\}^{\infty }_{k=1} \subset D_1$ such that

(2.19)\begin{equation} V(z^k)\rightarrow B=\inf\limits_{R^n}V(x)<0, \end{equation}

Set $\psi _k(x)=\eta (x-z^k)$. It is easy to see that

(2.20)\begin{equation} V(z^k)-\epsilon^k\psi_k(z^k)< B<0 \end{equation}

and

(2.21)\begin{equation} V(z^k)-\epsilon^k\psi_k(z^k)< V(x)-\epsilon^k\psi_k(x),\quad \mbox{for }x\in R^n\setminus B_l(z^k). \end{equation}

Define $V_k(x)=V(x)-\epsilon ^k\psi _k(x)$. It follows that there exists some point $\bar {z}^k\in B_l(z^k)$ such that

\[ V_k(\bar{z}^k)=\min\limits_{R^n}V_k(x)<0. \]

Similar to the proof of (2.16), by the second inequality of (2.10), we arrive at

(2.22)\begin{align} 0 & \leq (-\Delta)^{s}{V} (\bar{z}^k)-\bar{b}_2(\bar{z}^k){U}(\bar{z}^k)-\bar{c}_2(\bar{z}^k){V}(\bar{z}^k) \nonumber\\ & = (-\Delta)^{s}{V}_k (\bar{z}^k)-\bar{b}_2(\bar{z}^k){U}_k(\bar{z}^k)-\bar{c}_2(\bar{z}^k){V}_k(\bar{z}^k) \nonumber\\ & \quad - \bar{b}_2(\bar{z}^k){\varphi}_k(\bar{z}^k) \epsilon^k + (-\Delta)^{s}{\psi}_k (\bar{z}^k) \epsilon^k- \bar{c}_2(\bar{z}^k){\psi}_k(\bar{z}^k) \epsilon^k \nonumber\\ & \leq (\frac{C'}{l^{2s}}-\bar{c}_2(\bar{z}^k)){V}_k(\bar{z}^k) -\bar{b}_2(\bar{z}^k){U}_k(\bar{z}^k) \nonumber\\ & \quad - \bar{b}_2(\bar{z}^k){\varphi}_k(\bar{z}^k) \epsilon^k + (-\Delta)^{s}{\psi}_k (\bar{z}^k) \epsilon^k- \bar{c}_2(\bar{z}^k){\psi}_k(\bar{z}^k) \epsilon^k. \end{align}

For sufficiently small $l$ and sufficiently large $k$, we derive

(2.23)\begin{equation} 0 \leq \frac{C'}{l^{2s}}{V}_k(\bar{z}^k) -\bar{b}_2(\bar{z}^k){U}_k(\bar{z}^k)+o(\epsilon^k). \end{equation}

That is

(2.24)\begin{align} {V}_k(\bar{z}^k)& \geq C'l^{2s}\bar{b}_2(\bar{z}^k){U}_k(\bar{z}^k)+o(\epsilon^k) \nonumber\\ & \geq C'l^{2s}\bar{b}_2(\bar{z}^k){U}_k(\bar{x}^k)+o(\epsilon^k). \end{align}

By (2.17), we have

(2.25)\begin{equation} \bar{c}_1(\bar{x}^k){V}(\bar{x}^k)\leq \frac{C}{l^{2s}} {U}_k(\bar{x}^k)+ o(\epsilon^k). \end{equation}

Combining (2.24) with (2.25), we derive

(2.26)\begin{align} & C'l^{2s}\bar{c}_1(\bar{x}^k)\bar{b}_2(\bar{z}^k){U}_k(\bar{x}^k) \nonumber\\ & \quad \leq \bar{c}_1(\bar{x}^k){V}(\bar{z}^k) \nonumber\\ & \quad \leq \bar{c}_1(\bar{x}^k){V}(\bar{x}^k) \nonumber\\ & \quad \leq \frac{C}{l^{2s}} {U}_k(\bar{x}^k)+ o(\epsilon^k). \end{align}

This yields

(2.27)\begin{equation} C'\bar{b}_2(\bar{z}^k)\bar{c}_1(\bar{x}^k)\geq \frac{C}{l^{4s}}. \end{equation}

For sufficiently small $l$, (2.27) is impossible.

This completes the proof of lemma 2.2.

3. Monotonicity of solutions in bounded domains

In this section, we will verify theorem 1.1.

The proof of theorem 1.1 Consider the following system:

(3.1)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u (x)= f(u,\,v), \ & x\in G, \\ (-\Delta)^{s} v (x)= g(u,\,v), \ & x\in G, \\ u(x)=\varphi(x),\ v(x)=\psi(x),\ & x\in G^c, \end{array} \right. \end{equation}

where $G$ is a convex bounded domain in $x_n$ direction, and we denote the width of $G$ in $x_n$ direction as $d$.

First we introduce some basic notations. For any positive real number $\tau$, denote

\begin{align*} & u^{\tau}(x)=u(x',x_n+\tau),\quad v^{\tau}(x)=v(x',x_n+\tau), \\ & G_{\tau}=\{x-\tau e_n\,|\,x\in G\}, \end{align*}

here $x=(x_1,\,\dots,\,x_{n-1},\,x_n)=(x',\,x_n)$, $e_n=(0,\,\dots,\,0,\,1)$, and

\[ \Sigma_{\tau}=G\cap G_{\tau}. \]

Define

\[ \tilde{U}^{\tau}(x)=u^{\tau}(x)-u(x),\quad \tilde{V}^{\tau}(x)=v^{\tau}(x)-v(x). \]

This proof consists of two steps.

Step 1. For $0<\tau < d$ sufficiently large, we want to show

(3.2)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq 0,\quad \forall x\in G_{\tau}. \end{equation}

By the exterior conditions (P) of $u$ and $v$, it is easy to see that

(3.3)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq 0,\quad \forall x\in G_{\tau}\setminus\Sigma_{\tau}. \end{equation}

This implies that we only need to prove

(3.4)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq 0,\quad \forall x\in\Sigma_{\tau}. \end{equation}

Applying the mean value theorem to the first equation of (3.1), we have

(3.5)\begin{equation} (-\Delta)^{s}\tilde{U}^{\tau}(x)= f_u(\xi^{\tau}_1,\,v^{\tau})\tilde{U}^{\tau}(x)+f_v(u,\,\zeta^{\tau}_1)\tilde{V}^{\tau}(x),\quad x\in G, \end{equation}

where $\xi ^{\tau }_1$ is between $u$ and $u^{\tau }$ in $G$, and $\zeta ^{\tau }_1$ is between $v$ and $v^{\tau }$ in $G$.

Similarly, we have

(3.6)\begin{equation} (-\Delta)^{s}\tilde{V}^{\tau}(x)= g_u(\xi^{\tau}_2,\,v^{\tau})\tilde{U}^{\tau}(x)+g_v(u,\,\zeta^{\tau}_2)\tilde{V}^{\tau}(x),\quad x\in G, \end{equation}

where $\xi ^{\tau }_2$ is between $u$ and $u^{\tau }$ in $G$, and $\zeta ^{\tau }_2$ is between $v$ and $v^{\tau }$ in $G$.

Note that $\Sigma _{\tau }$ is a narrow region for $0<\tau < d$ sufficiently large. Applying lemma 2.1 to $\tilde {U}^{\tau }$ and $\tilde {V}^{\tau }$ with

\begin{align*} E& =\Sigma_{\tau},\quad b_1=f_u(\xi^{\tau}_1,\,v^{\tau}),\quad c_1=f_v(u,\,\zeta^{\tau}_1),\\ b_2& =g_u(\xi^{\tau}_2,\,v^{\tau}),\quad c_2=g_v(u,\,\zeta^{\tau}_2), \end{align*}

we derive that (3.4) is valid. We conclude that (3.2) must hold.

Step 2. Now we decrease $\tau$ as long as (3.2) holds to the limiting position. Define

\[ \tau_0=\inf\{\tau\,|\,\tilde{U}^{\tau}(x)\geq 0,\,\tilde{V}^{\tau}(x)\geq 0,x\in\Sigma_{\tau},\,0<\tau< d\}. \]

We want to prove

(3.7)\begin{equation} \tau_0=0.\end{equation}

If $\tau _0>0$, we can show that $G_{\tau _0}$ can be slid upward a little bit and we still have, for some small $\delta >0$ and $\tau \in (\tau _0-\delta,\,\tau _0)$,

(3.8)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq 0,\quad \forall\ x\in \Sigma_{\tau}. \end{equation}

This contradicts with the definition of $\tau _0$. Therefore, (3.7) holds. We postpone proving (3.8).

In fact, for $\tau _0>0$, we can show that

(3.9)\begin{equation} \tilde{U}^{\tau_0}(x)>0,\quad \tilde{V}^{\tau_0}(x)>0,\quad x\in \Sigma_{\tau_0}. \end{equation}

Otherwise, at least one of $\min \limits _{x\in \Sigma _{\tau _0}}\tilde {U}^{\tau _0}(x)$ and $\min \limits _{x\in \Sigma _{\tau _0}}\tilde {V}^{\tau _0}(x)$ are equal to zero. We may assume that, there exists a point $\bar {x}\in \Sigma _{\tau _0}$ such that

\[ \tilde{U}^{\tau_0}(\bar{x})=\min\limits_{x\in \Sigma_{\tau_0}}\tilde{U}^{\tau_0}(x)=0. \]

It follows from (3.5) that

(3.10)\begin{equation} (-\Delta)^s \tilde{U}^{\tau_0}(\bar{x})=f_v(u(\bar{x}),\,\zeta^{\tau_0}_1(\bar{x}))\tilde{V}^{\tau_0}(\bar{x}). \end{equation}

On the other hand, by the exterior condition (P) of $u$, we arrive at

(3.11)\begin{equation} (-\Delta)^s \tilde{U}^{\tau_0}(\bar{x})=C_{n,\,s}P.V.\int_{R^n}\frac{\tilde{U}^{\tau_0}(\bar{x})-\tilde{U}^{\tau_0}(y)}{|\bar{x}-y|^{n+2s}}\,{\rm d}y<0. \end{equation}

Combining (3.10) and (3.11) yields

\[ \tilde{V}^{\tau_0}(\bar{x})<0. \]

This is a contradiction. Hence (3.9) is valid. It follows that

(3.12)\begin{equation} \tilde{U}^{\tau_0}(x)>0,\quad \tilde{V}^{\tau_0}(x)>0,\quad \forall x\in \Sigma_{\tau_0}. \end{equation}

Next we can choose some closed $Q\subset \Sigma _{\tau _0}$ such that $\Sigma _{\tau _0}\setminus Q$ is a narrow region. Applying (3.12), we have

(3.13)\begin{equation} \tilde{U}^{\tau_0}(x)\geq c_0>0,\quad \tilde{V}^{\tau_0}(x)\geq c_0>0,\quad \forall x\in Q. \end{equation}

By the continuity of $\tilde {U}^{\tau }$ and $\tilde {V}^{\tau }$ in $\tau$, we obtain, for some small $\delta >0$ and $\tau \in (\tau _0-\delta,\,\tau _0)$,

(3.14)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq0,\quad \forall x\in Q. \end{equation}

Applying the exterior condition (P), we have, for some small $\delta >0$ and $\tau \in (\tau _0-\delta,\,\tau _0)$,

(3.15)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq0,\quad \forall x\in \Sigma^c_{\tau}. \end{equation}

It follows from lemma 2.1 that for some small $\delta >0$ and $\tau \in (\tau _0-\delta,\,\tau _0)$

(3.16)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq0,\quad \forall x\in \Sigma_{\tau}\setminus Q. \end{equation}

Combining (3.14), (3.15) and (3.16), we derive that, for some small $\delta >0$ and $\tau \in (\tau _0-\delta,\,\tau _0)$,

(3.17)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq0,\quad \forall x\in \Sigma_{\tau}. \end{equation}

This implies (3.8) holds. It follows that (3.7) must be true.

This completes the proof of theorem 1.1.

4. Monotonicity of solutions in unbounded domains

In this section, we study system (1.6). For convenience, we write down (1.6) again:

(4.1)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u (x)= f(u,\,v), \ & x\in \Omega, \\ (-\Delta)^{s} v (x)= g(u,\,v), \ & x\in \Omega, \\ u(x)=\varphi(x),\ v(x)=\psi(x),\ & x\in \Omega^c, \end{array} \right. \end{equation}

where $\Omega =\{x=(x',\,x_n)\in R^n\,|\,0< x_n< M\}$, $x'=(x_1,\,x_2,\,\dots,\,x_{n-1})$. We will verify theorem 1.2.

Proof of theorem 1.2 First we introduce some necessary notations. For any $0\leq \tau \leq M$, set

\[ u^{\tau}(x)=u(x',x_n+\tau),\quad v^{\tau}(x)=v(x',x_n+\tau). \]

Let

\[ \Omega_{\tau}=\{x-\tau e_n\,|\,x\in \Omega\}, \]

which is obtained by sliding $\Omega$ downward $\tau$ units in $x_n$ direction, $e_n=(0,\,0,\,\dots,\,0,\,1)$.

Set

\begin{align*} D_{\tau}& =\Omega\cap\Omega_{\tau},\\ U^{\tau}(x)& =u^{\tau}(x)-u(x),\quad V^{\tau}(x)=v^{\tau}(x)-v(x). \end{align*}

The proof consists of three steps.

Step 1. For $0<\tau < M$ sufficiently large, we want to show that

(4.2)\begin{equation} U^{\tau}(x)\geq 0,\quad V^{\tau}(x)\geq 0,\quad x\in \Omega_{\tau}. \end{equation}

Obviously,

\[ \Omega_{\tau}=D_{\tau}\cup (\Omega_{\tau}\cap R^n_{-}). \]

By the exterior condition (P) of $u$ and $v$, we get

(4.3)\begin{equation} U^{\tau}(x)\geq 0,\quad V^{\tau}(x)\geq 0,\quad x\in \Omega_{\tau}\cap R^n_{-}. \end{equation}

It is easy to see that $u^{\tau }(x)$ and $v^{\tau }(x)$ satisfy the PDEs (4.1). Combining with the mean value theorem, we obtain

(4.4)\begin{equation} (-\Delta)^{s}U^{\tau}(x)= f_u(\xi^{\tau}_1,\,v^{\tau})U^{\tau}(x)+f_v(u,\,\zeta^{\tau}_1)V^{\tau}(x),\quad x\in \Omega, \end{equation}

where $\xi ^{\tau }_1$ is between $u$ and $u^{\tau }$ in $\Omega$, and $\zeta ^{\tau }_1$ is between $v$ and $v^{\tau }$ in $\Omega$.

Similarly, we have

(4.5)\begin{equation} (-\Delta)^{s}V^{\tau}(x)= g_u(\xi^{\tau}_2,\,v^{\tau})U^{\tau}(x)+g_v(u,\,\zeta^{\tau}_2)V^{\tau}(x),\quad x\in \Omega, \end{equation}

where $\xi ^{\tau }_2$ is between $u$ and $u^{\tau }$ in $\Omega$, and $\zeta ^{\tau }_2$ is between $v$ and $v^{\tau }$ in $\Omega$.

For $\tau$ sufficiently close to $M$, $D_{\tau }$ is narrow region in $x_n$ direction. Applying the ‘narrow region principle for system on unbounded domains’ (lemma 2.2), we arrive at

(4.6)\begin{equation} U^{\tau}(x)\geq 0,\quad V^{\tau}(x)\geq 0,\quad \forall x\in D_{\tau}. \end{equation}

Combining (4.3) and (4.6), we derive that (4.2) must hold.

Step 2. (4.2) provides a starting point to carry out the sliding method. Now we decrease $\tau$ as long as (4.2) holds to the limiting position. Define

\[ \tau_0=\inf\{\tau\,|\,U^{\tau}(x)\geq 0,\,V^{\tau}(x)\geq 0,\,x\in D_{\tau},\,0<\tau< M\,\}. \]

We will show that

(4.7)\begin{equation} \tau_0=0.\end{equation}

Otherwise, suppose that $\tau _0>0$, we can show that $\Omega _{\tau }$ can be slid upward a little bit and we still have, for some small $\delta >0$,

(4.8)\begin{equation} U^{\tau}(x)\geq 0,\quad V^{\tau}(x)\geq 0,\quad \tau_0-\delta<\tau\leq \tau_0.\end{equation}

This is a contradiction with the definition of $\tau _0$. Then (4.7) holds. We delay to prove (4.8).

To prove (4.8), we first show that

(4.9)\begin{equation} \inf_{x\in D_{\tau_0}}U^{\tau_0}(x)>0,\quad \inf_{x\in D_{\tau_0}}V^{\tau_0}(x)>0.\end{equation}

If (4.9) is not true, then at least one of $\inf _{x\in D_{\tau _0}}U^{\tau _0}(x)$ and $\inf _{x\in D_{\tau _0}}V^{\tau _0}(x)$ is equal to zero. We may assume that

\[ \inf_{x\in D_{\tau_0}}U^{\tau_0}(x)=0. \]

Hence, there exists a sequence $\{x^k\}^{\infty }_{k=1}\subset D_{\tau _0}$ such that

(4.10)\begin{equation} U^{\tau_0}(x^k)\rightarrow 0,\quad \mbox{as }k\rightarrow \infty. \end{equation}

Set

(4.11)\begin{equation} \eta(x)=\left\{\begin{array}{@{}ll} ae^{\dfrac{1}{|x|^2-r}}, \ & |x|< r, \\ 0, \ & |x|\geq r, \end{array} \right. \end{equation}

choosing $a=e^{1/r}$ and $r=\frac {M-\tau _0}{2}$, such that $\eta (0)=\max \limits _{R^n}\eta (x)=1$.

Let $\varphi _k=\eta (x-x^k)$. There exists a positive sequence $\{\varepsilon _k\}$ such that

\[ U^{\tau_0}(x^k)-\varepsilon_k \varphi_k(x^k)<0 \]

with $\varepsilon _k\rightarrow 0,$ as $k\rightarrow \infty$.

For any $x\in D_{\tau _0}\setminus B_r(x^k)$, $U^{\tau _0}(x)\geq 0$ and $\varphi _k(x)=0$. It is easy to see that,

(4.12)\begin{equation} U^{\tau_0}(x^k)-\varepsilon^k\varphi_k(x^k)< U^{\tau_0}(x)-\varepsilon^k\varphi_k(x),\ \mbox{for }x\in D_{\tau_0}\setminus B_r(x^k), \end{equation}

where $B_r(x^k)=\{x\in R^n\,|\,|x-x^k|< r\}$.

It follows that there exists some point $\tilde {x}^k\in B_r(x^k)\cap D_{\tau _0}$ such that

(4.13)\begin{equation} U^{\tau_0}(\tilde{x}^k)-\varepsilon^k\varphi_k(\tilde{x}^k)=\min_{x\in D_{\tau_0}}(U^{\tau_0}(x)-\varepsilon^k\varphi_k(x))<0. \end{equation}

Combining (4.10) and (4.13) yields that

\[ U^{\tau_0}(x^k)\leq U^{\tau_0}(\tilde{x}^k)\leq U^{\tau_0}(x^k)-\varepsilon^k\varphi_k({x}^k)+\varepsilon^k\varphi_k(\tilde{x}^k). \]

Obviously, as $k\rightarrow \infty$,

(4.14)\begin{equation} U^{\tau_0}(\tilde{x}^k)\rightarrow 0.\end{equation}

By (4.4), we derive that, for $k$ sufficiently large,

(4.15)\begin{align} & (-\Delta)^s (U^{\tau_0}-\varepsilon_k\varphi_k)(\tilde{x}^k) \nonumber\\ & \quad = f_u(\xi^{\tau_0}_1(\tilde{x}^k),\,v^{\tau_0}(\tilde{x}^k))U^{\tau_0}(\tilde{x}^k) +f_v(u(\tilde{x}^k),\,\zeta^{\tau_0}_1(\tilde{x}^k))V^{\tau_0}(\tilde{x}^k)+ o(\varepsilon_k), \end{align}

where $\xi ^{\tau _0}_1(\tilde {x}^k)$ is between $u(\tilde {x}^k)$ and $u^{\tau _0}(\tilde {x}^k)$, and $\zeta ^{\tau _0}_1(\tilde {x}^k)$ is between $v(\tilde {x}^k)$ and $v^{\tau _0}(\tilde {x}^k)$.

On the other hand, employing the definition of the fractional Laplacian,

(4.16)\begin{align} & (-\Delta)^s (U^{\tau_0}-\varepsilon_k\varphi_k)(\tilde{x}^k) \nonumber\\ & \quad = C_{n,s} P.V.\int_{R^n} \frac{(U^{\tau_0}-\varepsilon_k \varphi_k)(\tilde{x}^k)-(U^{\tau_0}-\varepsilon_k \varphi_k)(y)}{|\tilde{x}^k-y|^{n+2s}}\,{\rm d}y \nonumber\\ & \quad \leq c \int_{B^c_r(\tilde{x}^k)} \frac{(U^{\tau_0}-\varepsilon_k \varphi_k)(\tilde{x}^k)-(U^{\tau_0}-\varepsilon_k \varphi_k)(y)}{|\tilde{x}^k-y|^{n+2s}}\,{\rm d}y \nonumber\\ & \quad \leq c \int_{B^c_r(0)} \frac{(U^{\tau_0}-\varepsilon_k\varphi_k)(\tilde{x}^k)-(U^{\tau_0}-\varepsilon_k\varphi_k)(y+\tilde{x}^k)}{|y|^{n+2s}}\,{\rm d}y \nonumber\\ & \quad \leq c \int_{B^c_r(0)} \frac{-U^{\tau_0}(y+\tilde{x}^k)}{|y|^{n+2s}}\,{\rm d}y. \end{align}

Set $u_k(x)=u(x+\tilde {x}^k)$, $U^{\tau }_k(x)=U^{\tau }(x+\tilde {x}^k)$. By Arzel$\grave {a}$–Ascoli theorem, we have

\[ u_k(x)\rightarrow u_{\infty}(x),\quad \mbox{as }k\rightarrow \infty,\quad \mbox{in }R^n. \]

Hence, as $k\rightarrow \infty$,

(4.17)\begin{equation} U^{\tau}_k(x)\rightarrow U^{\tau}_\infty(x)=u^{\tau}_{\infty}(x)-u_{\infty}(x),\quad x\in B^c_r(0).\end{equation}

Combining (4.15), (4.16) and (4.17), we deduce that, as $k\rightarrow \infty$,

(4.18)\begin{equation} \int_{B^c_r(0)} \frac{-U^{\tau_0}_{\infty}(y)}{|y|^{n+2s}}\,{\rm d}y \geq 0.\end{equation}

Obviously, (4.18) holds unless

(4.19)\begin{equation} U^{\tau_0}_{\infty}(y)\equiv 0,\quad y\in B^c_r(0). \end{equation}

By (4.19), we derive

(4.20)\begin{equation} u_{\infty}(x',x_n)=u_{\infty}(x',x_n+\tau_0)=\dots=u_{\infty}(x',x_n+m\tau_0) \end{equation}

for any $m\in {N}^+$.

Choosing $(x',\,x_n)\in \Omega$, and taking $m$ large enough such that $(x',\,x_n+m\tau _0)\in \Omega ^c$, we apply the exterior condition on $u$ to derive a contradiction with (4.20). Thus, (4.9) holds.

Choosing sufficiently large $K\subset D_{\tau _0}$ such that $D_{\tau _0}\setminus K$ is narrow in $x_n$ direction. Combining (4.9) with the continuity of $U^{\tau }$ and $V^{\tau }$ in $\tau$, we derive that, for some small $\delta >0$,

(4.21)\begin{equation} U^{\tau_0-\delta}(x)\geq 0,\quad V^{\tau_0-\delta}(x)\geq 0,\quad x\in K. \end{equation}

Meanwhile, applying the exterior condition (P), we have

(4.22)\begin{equation} U^{\tau_0-\delta}(x)\geq 0,\quad V^{\tau_0-\delta}(x)\geq 0,\quad x\in (D_{\tau_0-\delta})^{c}. \end{equation}

Employing lemma 2.2, we derive

(4.23)\begin{equation} U^{\tau_0-\delta}(x)\geq 0,\quad V^{\tau_0-\delta}(x)\geq 0,\quad x\in D_{\tau_0-\delta} \setminus K. \end{equation}

Combining (4.21) with (4.23), we obtain

(4.24)\begin{equation} U^{\tau_0-\delta}(x)\geq 0,\quad V^{\tau_0-\delta}(x)\geq 0,\quad x\in D_{\tau_0-\delta}. \end{equation}

This contradicts the definition of $\tau _0$. Hence, (4.7) is valid. We conclude that $u$ and $v$ are increasing in $x_n$ variable.

This completes the proof of theorem 1.2.