Stable solutions to double phase problems involving a nonlocal term

Abstract

In this paper, we study weak solutions, possibly unbounded and sign-changing, to the double phase problem

\begin{equation*}-\text{div} (|\nabla u|^{p-2} \nabla u + w(x)|\nabla u|^{q-2} \nabla u) = \left(\frac{1}{|x|^{N-\mu}}*f|u|^r\right) f(x)|u|^{r-2}u \quad\text{in}\ \mathbb{R}^N,\end{equation*}

where $q\ge p\ge2$, r > q, $0 \lt \mu \lt N$ and $w,f \in L^1_{\rm loc}(\mathbb{R}^N)$ are two non-negative functions such that $w(x) \le C_1|x|^a$ and $f(x) \ge C_2|x|^b$ for all $|x| \gt R_0$, where $R_0,C_1,C_2 \gt 0$ and $a,b\in\mathbb{R}$. Under some appropriate assumptions on p, q, r, µ, a, b and N, we prove various Liouville-type theorems for weak solutions which are stable or stable outside a compact set of $\mathbb{R}^N$. First, we establish the standard integral estimates via stability property to derive the non-existence results for stable weak solutions. Then, by means of the Pohožaev identity, we deduce the Liouville-type theorem for weak solutions which are stable outside a compact set.

1. Introduction

In the last decades, the stable and finite Morse index sign-changing solutions of weighted p-Laplace equations on unbounded domains of $\mathbb{R}^N$ have received a lot of attention (see e.g., [1–3, 6, 7, 9, 11, 12, 15, 16] and the references therein). The definition of stability is motivated by a phenomenon in physical sciences, which states that a system is in a stable state if it can recover from small perturbations. We refer to the monograph [5] for more discussions on the physical motivation and mathematical background of stable solutions.

Liouville theorems for stable solutions, which concern about nonexistence of this particular type of solutions, have drawn much attention in the last decade. In his celebrated article [6], Farina established a sharp Liouville theorem for stable classical solutions to the problem

(1)\begin{align} -\text{div}\left(\omega_1(x)\nabla u\right)=\omega_2(x)|u|^{q-1}u \quad \text{in}\ \mathbb{R}^N \end{align}

with $\omega_1(z)=\omega_2(z)\equiv1$ and q > 1. He showed that the problem does not admit any non-trivial stable C ² solution if and only if $1 \lt q \lt q_c(N), $ where

\begin{equation*} {q_c}(N): = \left\{ {\matrix{{ + \infty \;\;} \hfill & {{\rm{if N}} \le {\rm{10}},} \hfill \cr {{{{{(N - 2)}^2} - 4N + 8\sqrt {N - 1} } \over {(N - 2)(N - 10)}}\;\;} \hfill & {{\rm{if N}} \ge {\rm{11}}.} \hfill \cr } } \right.\end{equation*}

After that, the above results have been extended to the weighted case $\omega_1(x)\not\equiv1$ or $\omega_2(x)\not\equiv1$ in [1, 3, 16, 19]. In [3], under the restriction that the solutions are locally bounded, the authors presented the non-existence of non-trivial stable weak solutions of problem (1). Later, this restriction was removed in [19]. In [1], under various assumptions on $\omega_1(x)$ and $\omega_2(x)$, Cowan and Fazly established several Liouville-type theorems for stable positive classical solutions of problem (1). In particular, they examined a specific class of weights $\omega_1(x)= (|x|^2+1)^{\frac{\gamma_1}{2}}g(x)$ and $\omega_2(x)=(|x|^2+1)^{\frac{\gamma_2}{2}}g(z)$, where g is a positive function with a finite limit at $\infty$. For this class of weights, non-existence results are optimal.

Recently, Zhao [20] studied the non-existence of finite Morse index solution for the equation

(2)\begin{equation} -\Delta u= \left(|z|^{-\gamma} \ast |u|^{q+1}\right) \left|u\right|^{q-1}u \quad \mbox{in}\; \mathbb{R}^N, \; N \gt 2. \end{equation}

In [20], the author showed that problem (2) has no non-trivial solution with finite Morse index if $0 \lt \gamma \lt \min\{4, N\}$ and $1 \lt q \lt \frac{N+2-\gamma}{N-2} $. Notice that the right-hand side of Equation (2) is a non-local term which is usually referred to as the Hartree-type non-linearity in the literature. This kind of equation is usually called the Choquard-type equation since, in 1976, a similar equation as Equation (2) was used by P. Choquard to describe an electron trapped in its hole, in a certain approximation to Hartree–Fock theory of one component plasma [13]. In some contexts, equation of type (2) is also called the non-linear Schrödinger–Newton equation or the stationary Hartree equation. The second author [10] proved that this equation does not possess a positive solution for $1 \lt q \lt \frac{N+2-\gamma}{N-2}$ by using the moving plane method. In [9], with the help of Farina’s approach, the second author showed that Equation (2) has no non-trivial stable weak solution if $0 \lt \gamma \lt \min\{4, N\}$, N > 2 and q > 1. This phenomenon is quite different from that of the Lane–Emden equation studied by Farina [6], where such a result only holds for low exponents in high dimensions.

Liouville theorems for $C^{1,\alpha}$ solutions of the p-Laplace Hartree equation

\begin{equation*} -\Delta_p u = \left(\frac{1}{|x|^{N-\alpha}}*|u|^q\right) |u|^{q-2}u \quad\text{in}\ \mathbb{R}^N, \end{equation*}

was also examined by the second author [12]. He proved that if $2\le p \lt N$, $\max\{0,N-2p\} \lt \alpha \lt N$, $p \lt q \lt q_c$ and u is stable, then $u\equiv0$. Here q_c is a new critical exponent, which equals infinity when $\frac{N+\alpha}{N-p} \ge \frac{p+1}{2}$. He also showed that if $p \lt q \lt \frac{p(N+\alpha)}{2(N-p)}$ and u is stable outside a compact set or has a finite Morse index, then $u\equiv0$. The results in [12] cover the ones in [9, 20] when p = 2.

Besides the standard quasilinear operators, the so-called double phase problem

(3)\begin{equation} -\text{div} (|\nabla u|^{p-2} \nabla u + \gamma_1(x)|\nabla u|^{q-2} \nabla u) = \gamma_2(x)|u|^{r-1}u \quad\text{in}\ \mathbb{R}^N \end{equation}

gets a lot of attention in recent years. The operator on the left-hand side of Equation (3) is called the double phase operator since its behaviour switches between two different elliptic situations depending on the values of the weight function w. This kind of problem and the associated energy functionals arise in many applications. In the non-linear elasticity theory, the modulating coefficient w deforms the geometry of composites made of two different materials with distinct power hardening exponents q and p. Zhikov et al. [8, 21] used double phase functionals to describe models of strongly anisotropic materials in the context of homogenization. Double phase functionals also play an important role in the study of duality theory and the context of the Lavrentiev phenomenon [22].

Recently, the second author [11] obtained classification for stable sign-changing solutions to problem (3) as follows.

Theorem A. (see [11, Theorem 1]). Let u be a stable solution of Equation (3), where $q \ge p\ge2$, $r \gt q-1$ and $\gamma_1,\gamma_2 \in L^1_{\rm loc}(\mathbb{R}^N)$ are two non-negative functions such that $\gamma_1(x) \le C_1|x|^a$ and $\gamma_2(x) \ge C_2|x|^b$ for all $|x| \gt R_0$, with $R_0,C_1,C_2 \gt 0$ and $a,b\in\mathbb{R}$. Assume that

\begin{equation*} N \lt N^\sharp := \min\left\{\frac{p(\beta_0+r) + b(\beta_0+p-1)}{r-p+1}, \frac{(q-a)(\beta_0+r) + b(\beta_0+q-1)}{r-q+1}\right\}, \end{equation*}

where

\begin{equation*} \beta_0 := \frac{2r -q + 1 + 2\sqrt{r(r-q+1)}}{q-1}. \end{equation*}

Then $u \equiv 0$.

Theorem B. (see [11, Theorem 2]). Let u be a solution of Equation (3) with $\gamma_1(x) =|x|^a$ and $\gamma_2(x) =|x|^b$ such that u is stable outside a compact set, where $q \gt p\ge2$ and $r \gt q-1$. Assume furthermore that $|\nabla u|^{p-2} \nabla u + |x|^a|\nabla u|^{q-2} \nabla u \in W^{1,2}_{\rm loc}(\mathbb{R}^N,\mathbb{R}^N)$.

1. (i) If $\frac{N+b}{r+1} \gt \max\left\{\frac{N-p}{p}, \frac{N-q+a}{q}\right\}$, then $u \equiv 0$.

2. (ii) If $\frac{N+b}{r+1} = \max\left\{\frac{N-p}{p}, \frac{N-q+a}{q}\right\}$, then we have the identity

   \begin{equation*} \frac{N-p}{p} \int_{\mathbb{R}^N} |\nabla u|^p\, {\rm d}x + \frac{N-q+a}{q} \int_{\mathbb{R}^N} |x|^a |\nabla u|^q\, {\rm d}x = \frac{N+b}{r+1} \int_{\mathbb{R}^N} |x|^b |u|^{r+1} \,{\rm d}x \lt \infty. \end{equation*}

In this paper, we prove analogous results for the double phase problem involving a non-local term

(4)\begin{equation} -\text{div} (|\nabla u|^{p-2} \nabla u + w(x)|\nabla u|^{q-2} \nabla u) = \left(\frac{1}{|x|^{N-\mu}}*f|u|^r\right) f(x)|u|^{r-2}u \quad\text{in}\ \mathbb{R}^N, \end{equation}

where $q\ge p\ge2$, r > q, $0 \lt \mu \lt N$ and $w,f \in L^1_{\rm loc}(\mathbb{R}^N)$ are two non-negative functions such that $w(x) \le C_1|x|^a$ and $f(x) \ge C_2|x|^b$ for all $|x| \gt R_0$, where $R_0,C_1,C_2 \gt 0$, $a,b\in\mathbb{R}$ and $\max\{0,N+2(a-q-b)\} \lt \mu \lt N$. The main feature of problem (4) is that it combines the double phase phenomenon on the left-hand side and the non-local phenomenon on the right-hand side. This causes some difficulty, which makes the study of such a problem interesting. The existence of solutions to problem (4) in bounded domains was obtained recently in [17]. More precisely, Sun and Chang [17] studied the problem

(5)\begin{equation} \begin{cases} -\text{div} (|\nabla u|^{p-2} \nabla u + w(x)|\nabla u|^{q-2} \nabla u) = \left(\frac{1}{|x|^{N-\mu}}*|u|^r\right) |u|^{r-2}u &\text{in}\ \Omega,\\ u=0 & \text{on}\ \partial\Omega, \end{cases} \end{equation}

where Ω is a bounded domain of $\mathbb{R}^N$ and $1 \lt p \lt q \lt N$. Using the constrained variational method and Brouwer degree theory, they proved the existence of least energy nodal solutions to Equation (5) under a subcritical assumption on r. Unlike problem (5), it is usually unfavourable for elliptic problems in the whole space such as Equation (4) to have positive solutions under some subcritical assumption. In fact, our main Liouville-type theorem holds for problem (4), with r being less than some critical exponent r_c (see Theorem 1 below).

We recall functional settings for double phase problems. Let $H: \Omega \times [0, \infty) \to [0, \infty)$ be the function $H(x,t) = t^p + w(x) t^q$, where $\Omega\subset\mathbb{R}^N$ is a domain of $\mathbb{R}^N$. We define

\begin{equation*} \rho_H(u) = \int_{\Omega} H(x, |u|) {\rm d}x = \int_{\Omega} (|u|^p + w(x) |u|^q) {\rm d}x \end{equation*}

and

\begin{equation*} L^H(\Omega) = \{u:\Omega\to\mathbb{R} \mid u\ \text{is measurable and}\ \rho_H(u) \lt \infty\}, \end{equation*}

which is called the Musielak–Orlicz space. This space is equipped with the Luxemburg norm

\begin{equation*} \|u\|_H = \inf\left\{\tau \gt 0 \mid \rho_H\left(\frac{u}{\tau}\right) \le 1 \right\}. \end{equation*}

Then we define the Musielak–Orlicz Sobolev space

\begin{equation*} W^{1,H}(\Omega) = \left\{u \in L^H(\Omega) \mid |\nabla u| \in L^H(\Omega)\right\}, \end{equation*}

which admits the norm

\begin{equation*} \|u\|_{1,H} = \|\nabla u\|_H + \|u\|_H. \end{equation*}

As usual, we define $W^{1,H}_0(\Omega)$ as the closure of $C_c^1(\Omega)$ with respect to the norm in $W^{1,H}(\Omega)$. Moreover, we set

\begin{equation*} W^{1,H}_{\rm loc}(\Omega) = \left\{u:\Omega\to\mathbb{R} \mid u\varphi \in W^{1,H}_0(\Omega)\ \text{for all}\ \varphi \in C_c^1(\Omega)\right\}. \end{equation*}

Although the $C^{1,\alpha}_{\rm loc}$ regularity of the solutions for p-Laplace equations is well known (see, for instance, [4, 18]), the $C^{1,\alpha}_{\rm loc}$ regularity cannot be guaranteed for double phase problems due to the behaviour of the weight w. Therefore, it is more natural to work with the notion of weak local solutions as follows.

Definition 1. A function $u\in W^{1,H}_{\rm loc}(\mathbb{R}^N)$ is said to be a weak solution of Equation (4) if

\begin{equation*} \left(\frac{1}{|x|^{N-\mu}}*f|u|^r\right) f(x)|u|^{r-1} \in L^1_{\rm loc}(\mathbb{R}^N) \end{equation*}

and

(6)\begin{align} \int_{\mathbb{R}^N}& \left(|\nabla u|^{p-2} \nabla u + w(x)|\nabla u|^{q-2} \nabla u\right) \cdot \nabla\varphi \,{\rm d}x\\&\qquad = \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) \varphi(x) f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \end{align}

for all $\varphi \in C_c^1(\mathbb{R}^N)$. Furthermore, u is called a finite energy solution if

(7)\begin{equation} \int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x)|\nabla u|^q\right) {\rm d}x + \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r} f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \lt \infty. \end{equation}

Notice that condition (7) is also used in the literature to characterize Coulomb–Sobolev spaces, see [14].

Motivated by [2, 3, 11, 12, 15], in this paper, we are interested in Liouville theorems for stable and finite Morse index solutions of Equation (4), which are defined as follows.

Definition 2. A weak solution u of (4) is

• • stable if the quadratic form of energy functional associated to Equation (4) at u is non-negative, i.e.,

  \begin{align*} Q_u(\varphi) &:= \int_{\mathbb{R}^N} \left[|\nabla u|^{p-2} |\nabla\varphi|^2 + (p-2)|\nabla u|^{p-4} (\nabla u \cdot \nabla\varphi)^2\right] {\rm d}x\\ &\qquad+ \int_{\mathbb{R}^N} w(x)\left[|\nabla u|^{q-2} |\nabla\varphi|^2 + (q-2)|\nabla u|^{q-4} (\nabla u \cdot \nabla\varphi)^2\right] {\rm d}x\\ &\qquad - (r-1) \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} \varphi(x)^2 f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\qquad- r \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) \varphi(x) f(y)|u(y)|^{r-2} u(y) \varphi(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\ge 0 \end{align*}
  for all $\varphi \in C^1_c(\mathbb{R}^N)$,

• • stable outside a compact set $K\subset\mathbb{R}^N$ if $Q_u(\varphi) \ge 0$ for all $\varphi \in C_c^1(\mathbb{R}^N\setminus K)$,

• • has a Morse index equal to $k\ge0$ if k is the maximal dimension of a subspace X_k of $C_c^1(\mathbb{R}^N)$ such that $ Q_u(\varphi) \lt 0$ for all $\varphi \in X_k\setminus\{0\}$.

Remark 1. By Schwartz’s inequality, if u is a stable solution to Equation (4), then

(8)\begin{equation} \begin{split} &(p-1) \int_{\mathbb{R}^N} |\nabla u|^{p-2} |\nabla \varphi|^2\, {\rm d}x +(q-1) \int_{\mathbb{R}^N} w(x) |\nabla u|^{q-2} |\nabla \varphi|^2 \,{\rm d}x \\ &- (r-1) \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} \varphi(x)^2 f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &- r \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) \varphi(x) f(y)|u(y)|^{r-2} u(y) \varphi(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \ge 0 \end{split} \end{equation}

for all $\varphi \in C^1_c(\mathbb{R}^N)$. Moreover, Equations (6) and (8) hold for all $\varphi \in W^{1,H}_0(\Omega)$ by density arguments.

As far as we know, there are no non-existence results on stable and finite Morse index solutions for Equation (4) with $q\geq p\geq2$ and $w\not\equiv0$. In this paper, we will establish some Liouville-type theorems for such solutions. Our first result reads as follows.

Theorem 1. Assume that

1. (i) $2\le p \lt N$,

2. (ii) $p \le q \le p + \frac{pa}{N}$,

3. (iii) $\max\{0,N+2(a-q-b)\} \lt \mu \lt N$,

4. (iv) $ q \lt r \lt {r_c}: = \left\{ {\matrix{ { + \infty } \hfill & {{\rm{if}}\;{{N + \mu + 2b} \over {N + a - q}} \ge {{q + 1} \over 2},} \hfill \cr {{{(q - 1){{({\alpha _0} + 1)}^2}} \over {4{\alpha _0}}} + 1} \hfill & {{\rm{otherwise}},} \hfill \cr } } \right. $

   where

   \begin{align*} \alpha_0 &:= \frac{q(q+\mu-a+2b)-(N+\mu+2b)}{(q+1)(N+a-q)-2(N+\mu+2b)}\\ &\qquad + \frac{\sqrt{q(q-2)(N+\mu+2b)[\mu-N+2(q-a+b)] + (q+\mu-a+2b)^2}}{(q+1)(N+a-q)-2(N+\mu+2b)}. \end{align*}

Let u be a stable solution of Equation (4). Then $u\equiv0$.

Remark 2. Clearly, assumptions (i) and (ii) imply $a\ge0$ and $q\le p+a \lt N+a$.

When $\frac{N+\mu +2b}{N+a-q} \lt \frac{q+1}{2}$, one can verify that $\alpha_0 \gt 1$ and α ₀ is the largest solution of the equation

\begin{equation*} \frac{1}{2} \left[\frac{(q+\alpha-1)(N+\mu+2b)}{N +a- q} - (\alpha-1)\right]= \frac{(q-1)(\alpha+1)^2}{4\alpha} + 1, \end{equation*}

see the proof of Lemma 3 in the next section for more detail. We also remark that when $w=f\equiv1$ and q = p, Theorem 1 basically reduces to Theorem 1 in [12]. We stress, however, that only C ¹ solutions are considered in [12]. Hence, the statement of Liouville theorems in our paper is more general even in this specific case.

Remark 3. By assumption (iii), we have $\frac{N+\mu +2b}{N+a-q} \ge 2$. Hence, if $q\le3$, then $\frac{N+\mu +2b}{N+a-q} \geq \frac{q+1}{2}$ and the critical exponent r_c is infinity. This phenomenon has been observed by the second author [9] in the case $p=q=2$.

To prove Theorem 1, we follow the approach in [2, 11, 12]. As in these references, we test Equations (6) and (8) with suitable truncated functions of powers of u and exploit the Young inequality several times to obtain a Caccioppoli-type estimate (see Equation (16) below). Then we can control a term $L_R(u)$, which depends on the radius R > 0 and contains double integrals on $|\nabla u|$ and $|u|$, by its powers (see Equation (25)). Here a new idea is presented to show that this term goes to 0 as $R\to+\infty$, which implies $u\equiv0$. In order for the last step to work, we need that θ in the inequality (27) is negative. A sufficient condition to ensure that is that (ii), (iii) and (iv) hold. More precisely, assumptions (ii) and (iii) are rather technical, so that Equation (29) and Lemma 3 can be proved and used in the last step. However, we expect that r_c in (iv) is sharp, which is the case when $q\le3$ as we mentioned in Remark 3.

In this paper, we also study solutions stable outside a compact set of the problem

(9)\begin{equation} -\text{div} (|\nabla u|^{p-2} \nabla u + |x|^a|\nabla u|^{q-2} \nabla u) = \left(\frac{1}{|x|^{N-\mu}}*|\cdot|^b |u|^r\right) |x|^b |u|^{r-2}u \quad\text{in}\ \mathbb{R}^N, \end{equation}

which is a special case of problem (4) with $w(x)=|x|^a$ and $f(x)=|x|^b$. We will prove the following result by exploiting a Pohožaev-type identity.

Theorem 2. Assume $2\le p \lt N$, $p \le q \lt N+a$, $\max\{0,N+2(a-q-b)\} \lt \mu \lt N$ and r > q. Let u be a solution of Equation (9), which is stable outside a compact set such that $|\nabla u|^{p-2} \nabla u + |x|^a|\nabla u|^{q-2} \nabla u \in W^{1,2}_{\rm loc}(\mathbb{R}^N,\mathbb{R}^N)$.

1. (i) If $\frac{N+\mu +2b}{2r} \gt \max\left\{\frac{N-p}{p}, \frac{N+a-q}{q}\right\}$, then $u \equiv 0$.

2. (ii) If $\frac{N+\mu +2b}{2r} = \max\left\{\frac{N-p}{p}, \frac{N+a-q}{q}\right\}$, then we have the identity

   \begin{align*} & \frac{N-p}{p} \int_{\mathbb{R}^N} |\nabla u|^p\, {\rm d}x + \frac{N+a-q}{q} \int_{\mathbb{R}^N} |x|^a |\nabla u|^q \,{\rm d}x \\ & =\frac{N+\mu +2b}{2r} \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|x|^b |u(x)|^{r} |y|^b |u(y)|^{r}}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \lt \infty. \end{align*}

Remark 4. Let u be a solution with Morse index $k\ge1$. Then there exists a subspace $X_k := \text{span}\{\varphi_1,\dots,\varphi_k\} \subset C_c^1(\mathbb{R}^N)$ such that $Q_u(\varphi) \lt 0$ for all $\varphi \in X_k\setminus\{0\}$. Consequently, $Q_u(\varphi) \ge 0$ for all $\varphi \in C_c^1(\mathbb{R}^N\setminus K)$, where $K=\cup_{j=1}^k \text{supp}(\varphi_j)$. This means that u is stable outside the compact set K. Theorem 2 is, therefore, also valid for finite Morse index solutions.

The rest of this paper is devoted to the proof of our main results. In § 2, we prove Theorem 1 by exploiting the method of integral estimates with some ideas from the works [11, 12] of the second author. In § 3, we prove a Pohožaev-type identity and obtain some energy estimates. Then we use them to prove Theorem 2.

As usual, we use C to denote different positive constants whose values may change from line to line or even in the same line. Furthermore, we may append subscripts to C to specify its dependence on the subscript parameters. We also denote by B_R the ball centred at the origin with radius R > 0. We will drop notion ${\rm d}x$ in the integrals in $\mathbb{R}^N$ for brevity.

2. Liouville theorem for stable solutions

We will adopt some ideas from [2, 11, 12] in the proof of Theorem 1. Some non-trivial modifications are needed to deal with $W^{1,H}_{\rm loc}(\mathbb{R}^N)$ solutions and to overcome the combined effects of double phase and non-locality of Equation (4). We start with the following technical lemma, which will be used later in our integral estimates.

Lemma 3. Assume that $2\le q \lt N+a$, $\max\{0,N+2(a-q-b)\} \lt \mu \lt N$ and $q \lt r \lt r_c$, where r_c is given in Theorem 1. Then there exists $\alpha\ge1$ such that

\begin{equation*} \max\left\{\frac{(q-1)(\alpha+1)^2}{4\alpha} + 1, q+\frac{\alpha-1}{2}\right\} \lt r \lt \frac{1}{2} \left[\frac{(q+\alpha-1)(N+\mu+2b)}{N +a- q} - (\alpha-1)\right]. \end{equation*}

Proof. For $\alpha\ge1$, we define

\begin{equation*} f(\alpha) = \frac{(q-1)(\alpha+1)^2}{4\alpha} + 1, \qquad g(\alpha)= q+\frac{\alpha-1}{2}, \end{equation*}

\begin{equation*} h(\alpha)=\frac{1}{2} \left[\frac{(q+\alpha-1)(N+\mu+2b)}{N +a- q} - (\alpha-1)\right]. \end{equation*}

Since $N+2(a-q-b) \lt \mu$, we have $h(\alpha) \gt g(\alpha)$ for all $\alpha\ge1$ and

\begin{equation*} h(1)-f(1) = \frac{q(\mu+2b+2q-2a-N)}{2(N+a-q)} \gt 0. \end{equation*}

There are two cases.

Case 1: $\frac{N+\mu +2b}{N+a-q} \geq \frac{q+1}{2}$. In this case,

\begin{equation*} (h-f)^{\prime}(\alpha) = \frac{1}{2}\left(\frac{N+\mu +2b}{N+a-q} - \frac{q+1}{2}\right) + \frac{q-1}{4\alpha^2} \gt 0. \end{equation*}

Hence, $h(\alpha)-f(\alpha) \ge h(1)-f(1) \gt 0$ for all $\alpha\ge1$. Therefore, $\max\{f(\alpha), g(\alpha)\} \lt h(\alpha)$ for $\alpha\ge1$. On the other hand,

\begin{equation*} f(1) = g(1)=q\quad\text{and} \quad \lim_{\alpha\to+\infty} \max\{f(\alpha), g(\alpha)\}=+\infty. \end{equation*}

Thus, the claim follows from the continuity of $f,g,h$.

Case 2: $\frac{N+\mu +2b}{N+a-q} \lt \frac{q+1}{2}$. In this case,

\begin{equation*} \lim_{\alpha\to+\infty} \left(h(\alpha) - f(\alpha)\right)=-\infty. \end{equation*}

Combining this with $h(1)-f(1) \gt 0$, we deduce that $h(\alpha_0)=f(\alpha_0)$ for some $\alpha_0 \gt 1$. Moreover, α ₀ is given explicitly in Theorem 1. It is easy to see that α ₀ is the largest solution of the equation $h(\alpha)=f(\alpha)$. Furthermore, we have $h(\alpha) \gt f(\alpha)$ for all $1\le\alpha \lt \alpha_0$.

Hence, $\max\{f(\alpha), g(\alpha)\} \lt h(\alpha)$ for $1\le\alpha \lt \alpha_0$. On the other hand,

\begin{equation*} f(1) = g(1)=q\quad\text{and} \quad g(\alpha_0) \lt h(\alpha_0) = f(\alpha_0) = r_c. \end{equation*}

Then the conclusion follows from the continuity of $f,g,h$ as before.

We are in a position to prove the main result of this section, namely, Theorem 1.

Proof of Theorem 1

By Lemma 3, we can choose some $\alpha\ge1$ such that

(10)\begin{equation} \max\left\{\frac{(q-1)(\alpha+1)^2}{4\alpha} + 1, q+\frac{\alpha-1}{2}\right\} \lt r \lt \frac{1}{2} \left[\frac{(q+\alpha-1)(N+\mu+2b)}{N +a- q} - (\alpha-1)\right]. \end{equation}

We consider the following truncated functions for each $k\in\mathbb{N}$

\begin{equation*} a_k(t) = \begin{cases} |t|^{\frac{\alpha-1}{2}}t, & |t| \lt k,\\ k^{\frac{\alpha-1}{2}}t, & |t| \ge k,\\ \end{cases} \quad\text{and} \quad b_k(t) = \begin{cases} |t|^{\alpha-1}t, & |t| \lt k,\\ k^{\alpha-1}t, & |t| \ge k.\\ \end{cases} \end{equation*}

We observe that

(11)\begin{equation} \begin{split} &a_k(t)^2 \ge t b_k(t),\quad a_k^{\prime}(t)^2 \le \dfrac{(\alpha+1)^2}{4\alpha} b_k^{\prime}(t),\\ &|a_k(t)|^s a_k^{\prime}(t)^{2-s} + |b_k(t)|^s b_k^{\prime}(t)^{1-s} \le C_{\alpha,s} |t|^{\alpha + s - 1} \end{split} \end{equation}

for all $t\in\mathbb{R}$ and $s\ge2$. Moreover, using the fact $u\in W^{1,H}_{\rm loc}(\mathbb{R}^N)$, we see that $a_k(u), b_k(u) \in W^{1,H}_{\rm loc}(\mathbb{R}^N)$ for any $k \in \mathbb{N}$.

Let $\beta\ge q$, $\varepsilon\in(0,1)$ and $\psi\in C_c^1(\mathbb{R}^N)$ be such that $0\le\psi\le1$ in $\mathbb{R}^N$. Testing Equation (6) with $\varphi=b_k(u)\psi^\beta$, we have

(12)\begin{equation} \begin{aligned} &\int_{\mathbb{R}^N}|\nabla u|^p b_k^{\prime}(u) \psi^\beta + \beta \int_{\mathbb{R}^N}|\nabla u|^{p-2} b_k(u) \psi^{\beta-1} \nabla u \cdot \nabla\psi \\ &\qquad+ \int_{\mathbb{R}^N}w(x) |\nabla u|^q b_k^{\prime}(u) \psi^\beta + \beta \int_{\mathbb{R}^N}w(x) |\nabla u|^{q-2} b_k(u) \psi^{\beta-1} \nabla u \cdot \nabla\psi \\ &= \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) b_k(u(x))\psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x{\rm d}y. \end{aligned} \end{equation}

We estimate the second term by using Young’s inequality as follows:

\begin{align*} &-\beta \int_{\mathbb{R}^N}w(x) |\nabla u|^{q-2} b_k(u) \psi^{\beta-1} \nabla u \cdot \nabla\psi \\ &\le \beta\int_{\mathbb{R}^N}w(x) |\nabla u|^{q-1} |b_k(u)| \psi^{\beta-1} |\nabla\psi| \\ &\le \int_{\mathbb{R}^N}\left\{\varepsilon\left(w(x)^{\frac{q-1}{q}} |\nabla u|^{q-1} b_k^{\prime}(u)^\frac{q-1}{q} \psi^\frac{(q-1)\beta}{q} \right)^{\frac{q}{q-1}}\right. \\&\qquad +\left. C_\varepsilon\left(w(x)^{\frac{1}{q}} |b_k(u)| b_k^{\prime}(u)^\frac{1-q}{q} \psi^\frac{\beta-q}{q} |\nabla\psi| \right)^q\right\} \\ &\le \varepsilon \int_{\mathbb{R}^N}w(x) |\nabla u|^q b_k^{\prime}(u) \psi^\beta + C_\varepsilon\int_{\mathbb{R}^N}w(x) |b_k(u)|^q b_k^{\prime}(u)^{1-q} \psi^{\beta-q} |\nabla\psi|^q. \end{align*}

In the same way, we obtain

\begin{align*} -\beta \int_{\mathbb{R}^N}|\nabla u|^{p-2}& b_k(u) \psi^{\beta-1} \nabla u \cdot \nabla\psi \le \varepsilon \int_{\mathbb{R}^N}|\nabla u|^p b_k^{\prime}(u) \psi^\beta\\& + C_\varepsilon\int_{\mathbb{R}^N} |b_k(u)|^p b_k^{\prime}(u)^{1-p} \psi^{\beta-p} |\nabla\psi|^p. \end{align*}

Therefore, Equation (12) leads to

(13)\begin{align} \begin{split} &(1-\varepsilon) \int_{\mathbb{R}^N}\left(|\nabla u|^p + w(x) |\nabla u|^q\right) b_k^{\prime}(u) \psi^\beta \\ &\le \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) b_k(u(x))\psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \\ &\qquad + C_\varepsilon\int_{\mathbb{R}^N} |b_k(u)|^p b_k^{\prime}(u)^{1-p} \psi^{\beta-p} |\nabla\psi|^p + C_\varepsilon\int_{\mathbb{R}^N}w(x) |b_k(u)|^q b_k^{\prime}(u)^{1-q} \psi^{\beta-q} |\nabla\psi|^q \\ &\le \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) b_k(u(x))\psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \\ &\qquad + C_\varepsilon\int_{\mathbb{R}^N}|u|^{\alpha+p-1} \psi^{\beta-p} |\nabla\psi|^p + C_\varepsilon\int_{\mathbb{R}^N}w(x) |u|^{\alpha+q-1} \psi^{\beta-q} |\nabla\psi|^q, \end{split} \end{align}

where Equation (11) has been used in the last estimate.

Now we use $\varphi = a_k(u) \psi^{\frac{\beta}{2}}$ as a test function in Equation (8) and take into account the inequality

\begin{equation*} |x+y|^2 \le (1+\delta)|x|^2 + C_\delta |y|^2 \quad\text{for}\ x,y\in\mathbb{R}^N,\ \delta \gt 0. \end{equation*}

We obtain

(14)\begin{equation} \begin{split} &(r-1) \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} a_k(u(x))^2 \psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\!\!\!\!\!\qquad + r \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) a_k(u(x)) \psi^{\frac{\beta}{2}}(x) f(y)|u(y)|^{r-2} u(y) a_k(u(y)) \psi^{\frac{\beta}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \\ &\le \left(p-1 + \frac{\varepsilon}{2}\right) \int_{\mathbb{R}^N}|\nabla u|^p a_k^{\prime}(u)^2 \psi^\beta + A_\varepsilon\int_{\mathbb{R}^N}|\nabla u|^{p-2} a_k(u)^2 \psi^{\beta-2} |\nabla\psi|^2 \\ &\qquad+\left(q-1 + \frac{\varepsilon}{2}\right) \int_{\mathbb{R}^N}w(x) |\nabla u|^q a_k^{\prime}(u)^2 \psi^\beta + B_\varepsilon\int_{\mathbb{R}^N}w(x) |\nabla u|^{q-2} a_k(u)^2 \psi^{\beta-2} |\nabla\psi|^2. \end{split} \end{equation}

If q > 2, we can apply Young’s inequality to deduce

\begin{align*} &B_\varepsilon \int_{\mathbb{R}^N}w(x) |\nabla u|^{q-2} a_k(u)^2 \psi^{\beta-2} |\nabla\psi|^2 \\ &\le \int_{\mathbb{R}^N}\left\{\frac{\varepsilon}{2}\left(w(x)^{\frac{q-2}{q}} |\nabla u|^{q-2} a_k^{\prime}(u)^\frac{2(q-2)}{q} \psi^\frac{(q-2)\beta}{q}\right)^\frac{q}{q-2}\right.\\ &\qquad + \left.C_\varepsilon\left(w(x)^{\frac{2}{q}} a_k(u)^2 a_k^{\prime}(u)^\frac{2(2-q)}{q} \psi^\frac{2(\beta-q)}{q} |\nabla\psi|^2\right)^{\frac{q}{2}}\right\} \\ &= \frac{\varepsilon}{2} \int_{\mathbb{R}^N}w(x) |\nabla u|^q a_k^{\prime}(u)^2 \psi^\beta + C_\varepsilon\int_{\mathbb{R}^N}w(x) |a_k(u)|^q a_k^{\prime}(u)^{2-q} \psi^{\beta-q} |\nabla\psi|^{q} . \end{align*}

Similarly, if p > 2, we have

\begin{align*} &A_\varepsilon \int_{\mathbb{R}^N}|\nabla u|^{p-2} a_k(u)^2 \psi^{\beta-2} |\nabla\psi|^2\\ &\le \frac{\varepsilon}{2} \int_{\mathbb{R}^N}|\nabla u|^p a_k^{\prime}(u)^2 \psi^\beta + C_\varepsilon\int_{\mathbb{R}^N} |a_k(u)|^p a_k^{\prime}(u)^{2-p} \psi^{\beta-p} |\nabla\psi|^{p}. \end{align*}

Substituting these two estimates into Equation (14), we obtain

\begin{align*} &(r-1) \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} a_k(u(x))^2 \psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\!\!\!\!\!\qquad + r \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) a_k(u(x)) \psi^{\frac{\beta}{2}}(x) f(y)|u(y)|^{r-2} u(y) a_k(u(y)) \psi^{\frac{\beta}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\le (p-1+\varepsilon) \int_{\mathbb{R}^N}|\nabla u|^p a_k^{\prime}(u)^2 \psi^\beta + (q-1+\varepsilon) \int_{\mathbb{R}^N}w(x) |\nabla u|^q a_k^{\prime}(u)^2 \psi^\beta \\ &\qquad + C_\varepsilon\int_{\mathbb{R}^N} |a_k(u)|^p a_k^{\prime}(u)^{2-p} \psi^{\beta-p} |\nabla\psi|^{p} + C_\varepsilon\int_{\mathbb{R}^N}w(x) |a_k(u)|^q a_k^{\prime}(u)^{2-q} \psi^{\beta-q} |\nabla\psi|^{q}. \end{align*}

Notice that the above inequality also holds in the case p = 2 or q = 2.

Taking into account $q\ge p$ and Equation (11), we derive

(15)\begin{equation} \begin{split} &(r-1) \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} a_k(u(x))^2 \psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\!\!\!\!\!\qquad + r \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) a_k(u(x)) \psi^{\frac{\beta}{2}}(x) f(y)|u(y)|^{r-2} u(y) a_k(u(y)) \psi^{\frac{\beta}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\le (q-1+\varepsilon) \int_{\mathbb{R}^N} \left( |\nabla u|^p + w(x) |\nabla u|^q\right) a_k^{\prime}(u)^2 \psi^\beta \\ &\qquad + C_\varepsilon\int_{\mathbb{R}^N}|u|^{\alpha+p-1} \psi^{\beta-p} |\nabla\psi|^p + C_\varepsilon\int_{\mathbb{R}^N}w(x) |u|^{\alpha+q-1} \psi^{\beta-q} |\nabla\psi|^q. \end{split} \end{equation}

Using Equation (11), from Equations (13) and (15), we deduce

\begin{align*} &(r-1) \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} a_k(u(x))^2 \psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\!\!\!\!\!\qquad + r \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) a_k(u(x)) \psi^{\frac{\beta}{2}}(x) f(y)|u(y)|^{r-2} u(y) a_k(u(y)) \psi^{\frac{\beta}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\le \frac{(q-1+\varepsilon)(\alpha+1)^2}{4\alpha} \int_{\mathbb{R}^N} \left( |\nabla u|^p + w(x) |\nabla u|^q\right) b_k^{\prime}(u) \psi^\beta \\ &\qquad + C_\varepsilon\int_{\mathbb{R}^N}|u|^{\alpha+p-1} \psi^{\beta-p} |\nabla\psi|^p + C_\varepsilon\int_{\mathbb{R}^N}w(x) |u|^{\alpha+q-1} \psi^{\beta-q} |\nabla\psi|^q\\ &\le \frac{(q-1+\varepsilon)(\alpha+1)^2}{4\alpha(1-\varepsilon)} \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) b_k(u(x))\psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \\ &\qquad + C_\varepsilon\int_{\mathbb{R}^N}|u|^{\alpha+p-1} \psi^{\beta-p} |\nabla\psi|^p + C_\varepsilon\int_{\mathbb{R}^N}w(x) |u|^{\alpha+q-1} \psi^{\beta-q} |\nabla\psi|^q\\ &\le \frac{(q-1+\varepsilon)(\alpha+1)^2}{4\alpha(1-\varepsilon)} \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} a_k(u(x))^2\psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \\ &\qquad + C_\varepsilon\int_{\mathbb{R}^N}|u|^{\alpha+p-1} \psi^{\beta-p} |\nabla\psi|^p + C_\varepsilon\int_{\mathbb{R}^N}w(x) |u|^{\alpha+q-1} \psi^{\beta-q} |\nabla\psi|^q. \end{align*}

Therefore,

\begin{align*} &D_\varepsilon \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} a_k(u(x))^2 \psi(x)^\beta f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\!\!\!\!\!\qquad + r \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) a_k(u(x)) \psi^{\frac{\beta}{2}}(x) f(y)|u(y)|^{r-2} u(y) a_k(u(y)) \psi^{\frac{\beta}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ & \le C_\varepsilon\int_{\mathbb{R}^N}|u|^{\alpha+p-1} \psi^{\beta-p} |\nabla\psi|^p + C_\varepsilon\int_{\mathbb{R}^N}w(x) |u|^{\alpha+q-1} \psi^{\beta-q} |\nabla\psi|^q, \end{align*}

where

\begin{equation*} D_\varepsilon := r -1 - \frac{(q-1+\varepsilon)(\alpha+1)^2}{4\alpha(1-\varepsilon)}. \end{equation*}

From Equation (10), we have $\lim\limits_{\varepsilon\to 0^+} D_\varepsilon = r -1 - \frac{(q-1)(\alpha+1)^2}{4\alpha} \gt 0$. Hence, we can and do fix some ɛ > 0 such that $D_\varepsilon \gt 0$. We also choose $\beta=q$. Then

\begin{align*} & \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) a_k(u(x)) \psi^{\frac{q}{2}}(x) f(y)|u(y)|^{r-2} u(y) a_k(u(y)) \psi^{\frac{q}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ & \le C \int_{\mathbb{R}^N}|u|^{\alpha+p-1} \psi^{q-p} |\nabla\psi|^p + C \int_{\mathbb{R}^N}w(x) |u|^{\alpha+q-1} |\nabla\psi|^q. \end{align*}

Combining this with Equation (13) and using Equation (11) again, we can add one more term to the left-hand side of the above inequality as follows

\begin{align*} &\int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x) |\nabla u|^q\right) b_k^{\prime}(u) \psi^q\\ &\qquad + \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r-2} u(x) a_k(u(x)) \psi^{\frac{q}{2}}(x) f(y)|u(y)|^{r-2} u(y) a_k(u(y)) \psi^{\frac{q}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\le C \int_{\mathbb{R}^N}|u|^{\alpha+p-1} \psi^{q-p} |\nabla\psi|^p + C \int_{\mathbb{R}^N}w(x) |u|^{\alpha+q-1} |\nabla\psi|^q. \end{align*}

Letting $k\to\infty$, by Fatou’s lemma, we deduce

(16)\begin{equation} \begin{split} &\int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x) |\nabla u|^q\right) |u|^{\alpha-1} \psi^q \\ &\qquad + \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r+\frac{\alpha-1}{2}} \psi^{\frac{q}{2}}(x) f(y)|u(y)|^{r+\frac{\alpha-1}{2}} \psi^{\frac{q}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\le C \int_{\mathbb{R}^N}|u|^{\alpha+p-1} \psi^{q-p} |\nabla\psi|^p + C \int_{\mathbb{R}^N}w(x) |u|^{\alpha+q-1} |\nabla\psi|^q. \end{split} \end{equation}

Now we choose $\psi = \eta_R^m$, where $R \gt R_0$ and $\eta_R \in C^1_c(\mathbb{R}^N)$ satisfies $0\le\eta_R\le1$ and

(17)\begin{equation} \quad \eta_R=1\ \text{in}\ B_R, \quad \eta_R=0 \text{in}\ \mathbb{R}^N\setminus B_{2R}, \quad |\nabla \eta_R|\le \frac{C}{R}\ \text{in}\ B_{2R}\setminus B_R, \end{equation}

and the positive integer m is taken sufficiently large such that

(18)\begin{equation} \min\left\{\frac{(qm-p)(2r+\alpha-1)}{\alpha+p-1}, \frac{q(m-1)(2r+\alpha-1)}{\alpha+q-1}\right\} \ge m q. \end{equation}

Then Equation (16) becomes

(19)\begin{equation} \begin{split} &\int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x) |\nabla u|^q\right) |u|^{\alpha-1} \eta_R^{qm} \\ &\qquad + \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r+\frac{\alpha-1}{2}} \eta_R^{\frac{qm}{2}}(x) f(y)|u(y)|^{r+\frac{\alpha-1}{2}} \eta_R^{\frac{qm}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &\le C \int_{B_{2R}\setminus B_R}|u|^{\alpha+p-1} \eta_R^{qm-p} |\nabla\eta_R|^p + C \int_{B_{2R}\setminus B_R}w(x) |u|^{\alpha+q-1} \eta_R^{q(m-1)} |\nabla\eta_R|^q. \end{split} \end{equation}

We have

(20)\begin{equation} \begin{aligned} &\int_{B_{2R}\setminus B_R} |u|^{\alpha+p-1} \eta_R^{qm-p} |\nabla \eta_R|^p \\ &\le \left(\int_{B_{2R}\setminus B_R} f(x)|u|^{r+\frac{\alpha-1}{2}} \eta_R^\frac{qm}{2} \right)^\frac{2(\alpha+p-1)}{2r+\alpha-1}\\ &\qquad \times \left(\int_{B_{2R}\setminus B_R} f(x)^{-\frac{2(\alpha+p-1)}{2r-2p-\alpha+1}}|\nabla \eta_R|^\frac{p(2r+\alpha-1)}{2r-2p-\alpha+1} \right)^\frac{2r-2p-\alpha+1}{2r+\alpha-1}\\ &\le C R^{\frac{N(2r-2p-\alpha+1)}{2r+\alpha-1}-\frac{2b(\alpha+p-1)}{2r+\alpha-1}-p}\left(\int_{B_{2R}\setminus B_R} f(x) |u|^{r+\frac{\alpha-1}{2}} \eta_R^\frac{qm}{2} \right)^\frac{2(\alpha+p-1)}{2r+\alpha-1} \end{aligned} \end{equation}

and

(21)\begin{equation} \begin{aligned} &\int_{B_{2R}\setminus B_R}w(x) |u|^{\alpha+q-1} \eta_R^{q(m-1)} |\nabla \eta_R|^q \\ &\le \left(\int_{B_{2R}\setminus B_R} f(x)|u|^{r+\frac{\alpha-1}{2}} \eta_R^\frac{qm}{2} \right)^\frac{2(\alpha+q-1)}{2r+\alpha-1}\\ &\qquad \times \left(\int_{B_{2R}\setminus B_R} w(x)^\frac{(2r+\alpha-1)}{2r-2q-\alpha+1}f(x)^{-\frac{2(\alpha+q-1)}{2r-2q-\alpha+1}}|\nabla \eta_R|^\frac{q(2r+\alpha-1)}{2r-2q-\alpha+1} \right)^\frac{2r-2q-\alpha+1}{2r+\alpha-1}\\ &\le C R^{\frac{N(2r-2q-\alpha+1)}{2r+\alpha-1}-\frac{2b(\alpha+q-1)}{2r+\alpha-1}-q+a}\left(\int_{B_{2R}\setminus B_R} f(x) |u|^{r+\frac{\alpha-1}{2}} \eta_R^\frac{qm}{2} \right)^\frac{2(\alpha+q-1)}{2r+\alpha-1}, \end{aligned} \end{equation}

where we have applied Hölder’s inequality in Equations (20) and (21). Notice that we could do that because $2r-2p-\alpha+1\ge2r-2q-\alpha+1 \gt 0$ due to Equation (10).

Combining Equations (20) and (21), we obtain

(22)\begin{equation} \begin{split} & \int_{B_{2R}\setminus B_R}|u|^{\alpha+p-1} \eta_R^{qm-p} |\nabla\eta_R|^p + \int_{B_{2R}\setminus B_R}w(x) |u|^{\alpha+q-1} \eta_R^{q(m-1)} |\nabla\eta_R|^q\\ & \le C R^{\frac{N(2r-2p-\alpha+1)}{2r+\alpha-1}-\frac{2b(\alpha+p-1)}{2r+\alpha-1}-p}\left(\int_{\mathbb{R}^N} f(x)|u|^{r+\frac{\alpha-1}{2}} \eta_R^\frac{qm}{2} \right)^\frac{2(\alpha+p-1)}{2r+\alpha-1} \\ & \quad + C R^{\frac{N(2r-2q-\alpha+1)}{2r+\alpha-1}-\frac{2b(\alpha+q-1)}{2r+\alpha-1}-q+a}\left(\int_{\mathbb{R}^N} f(x) |u|^{r+\frac{\alpha-1}{2}} \eta_R^\frac{qm}{2} \right)^\frac{2(\alpha+q-1)}{2r+\alpha-1}. \end{split} \end{equation}

On the other hand,

(23)\begin{equation} \begin{split} & \left(\int_{\mathbb{R}^N} f(x) |u|^{r+\frac{\alpha-1}{2}} \eta_R^\frac{qm}{2} \right)^2 \\ &= \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} f(x) |u(x)|^{r+\frac{\alpha-1}{2}} \eta_R(x)^\frac{qm}{2} f(y)|u(y)|^{r+\frac{\alpha-1}{2}} \eta_R(y)^\frac{qm}{2} {\rm d}x\,{\rm d}y\\ & \le C R^{N-\mu} \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r+\frac{\alpha-1}{2}} \eta_R(x)^\frac{qm}{2} f(y)|u(y)|^{r+\frac{\alpha-1}{2}} \eta_R(y)^\frac{qm}{2}}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y. \end{split} \end{equation}

Setting

\begin{align*} L_R(u) &:= \int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x) |\nabla u|^q\right) |u|^{\alpha-1} \eta_R^{qm} \\ &\qquad + \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r+\frac{\alpha-1}{2}} \eta_R^{\frac{qm}{2}}(x) f(y)|u(y)|^{r+\frac{\alpha-1}{2}} \eta_R^{\frac{qm}{2}}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y. \end{align*}

From Equations (22) and (23), we deduce

(24)\begin{align} \begin{split} & \int_{B_{2R}\setminus B_R}|u|^{\alpha+p-1} \eta_R^{qm-p} |\nabla\eta_R|^p + \int_{B_{2R}\setminus B_R}w(x) |u|^{\alpha+q-1} \eta_R^{q(m-1)} |\nabla\eta_R|^q\\ & \le \; C R^{\frac{N(2r-2p-\alpha+1)}{2r+\alpha-1}-\frac{2b(\alpha+p-1)}{2r+\alpha-1}-p} R^\frac{(N-\mu)(\alpha+p-1)}{2r+\alpha-1} [L_R(u)]^\frac{\alpha+p-1}{2r+\alpha-1} \\ &\qquad + C R^{\frac{N(2r-2q-\alpha+1)}{2r+\alpha-1}-\frac{2b(\alpha+q-1)}{2r+\alpha-1}-q+a} R^\frac{(N-\mu)(\alpha+q-1)}{2r+\alpha-1} [L_R(u)]^\frac{\alpha+q-1}{2r+\alpha-1} \\ & = C R^{N-p-\frac{(p+\alpha-1)(N+\mu+2b)}{2r+\alpha-1}} [L_R(u)]^\frac{\alpha+p-1}{2r+\alpha-1} \\&\qquad + C R^{N+a-q-\frac{(q+\alpha-1)(N+\mu+2b)}{2r+\alpha-1}} [L_R(u)]^\frac{\alpha+q-1}{2r+\alpha-1}. \end{split} \end{align}

Substituting Equation (24) into Equation (19), we obtain

(25)\begin{align} L_R(u) \le C& R^{N-p-\frac{(p+\alpha-1)(N+\mu+2b)}{2r+\alpha-1}} [L_R(u)]^\frac{\alpha+p-1}{2r+\alpha-1}\\&\qquad + C R^{N+a-q-\frac{(q+\alpha-1)(N+\mu+2b)}{2r+\alpha-1}} [L_R(u)]^\frac{\alpha+q-1}{2r+\alpha-1}. \end{align}

We claim that

(26)\begin{equation} \lim_{R\to+\infty} L_R(u) = 0. \end{equation}

By contradiction, assume that there exists a sequence $R_n\to+\infty$ such that $L_{R_n}(u) \ge c$ for some c > 0. Then Equation (25) implies

\begin{equation*} L_{R_n}(u) \le C R_n^\theta [L_{R_n}(u)]^\frac{\alpha+q-1}{2r+\alpha-1}, \end{equation*}

i.e.,

(27)\begin{equation} [L_{R_n}(u)]^\frac{2r-q}{2r+\alpha-1} \le C R_n^\theta, \end{equation}

where

\begin{equation*} \theta := \max\left\{N-p-\frac{(p+\alpha-1)(N+\mu+2b)}{2r+\alpha-1}, N+a-q-\frac{(q+\alpha-1)(N+\mu+2b)}{2r+\alpha-1}\right\}. \end{equation*}

Notice that

(28)\begin{equation} \frac{1}{2} \left[\frac{(q+\alpha-1)(N+\mu+2b)}{N +a- q} - (\alpha-1)\right] \le \frac{1}{2} \left[\frac{(p+\alpha-1)(N+\mu+2b)}{N - p} - (\alpha-1)\right]. \end{equation}

Indeed, Equation (28) is equivalent to

(29)\begin{equation} qN - p(N+a) \le (a+p-q)(\alpha-1). \end{equation}

By assumption (ii), we have $a+p-q \ge p + \frac{pa}{N} - q \ge 0$. Hence, Equation (29) holds since the left-hand side of Equation (29) is non-positive and the right-hand side is non-negative.

Combining Equation (28) with Equation (10), we deduce

\begin{equation*} \theta \lt 0. \end{equation*}

Hence, Equation (27) implies $L_{R_n}(u) \to 0$, a contradiction. Therefore, Equation (26) holds, which means

\begin{equation*} \int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x) |\nabla u|^q\right) |u|^{\alpha-1} + \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r+\frac{\alpha-1}{2}} f(y)|u(y)|^{r+\frac{\alpha-1}{2}}}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y = 0. \end{equation*}

This only happens if u = 0 in $\mathbb{R}^N$.

3. Liouville theorem for solutions which are stable outside a compact set

As we mentioned in the introduction section, a Pohožaev-type identity will be exploited in the proof of Theorem 2. To this end, we will point out that the solutions which are stable outside a compact set are finite energy solutions.

Lemma 4. Assume that $2\le p \lt N$, $p \le q \lt N+a$, $\max\{0,N+2(a-q-b)\} \lt \mu \lt N$ and r > q. Let u be a solution of Equation (4), which is stable outside a compact set. If

(30)\begin{equation}\frac{N+\mu +2b}{2r} \ge \max\left\{\frac{N-p}{p}, \frac{N+a-q}{q}\right\}, \end{equation}

then u has finite energy, that is

\begin{equation*} \int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x)|\nabla u|^q\right) + \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r} f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \lt \infty. \end{equation*}

Proof. We assume that u is stable outside the compact set $K \subset \mathbb{R}^N$. Let $R_0 \gt 0$ be such that $K\subset B_{R_0}$ and $\phi_R \in C^1_c(\mathbb{R}^N)$ satisfies $0\le\phi_R\le1$ and

\begin{equation*} \phi_R=0\ \text{in}\ B_{R_0} \cup \left(\mathbb{R}^N\setminus B_{2R}\right), \quad \phi_R=1\ \text{in}\ B_R\setminus B_{R_0+1}, \quad |\nabla \phi_R|\le \frac{C}{R}\ \text{in}\ B_{2R} \setminus B_R. \end{equation*}

Notice that to obtain Equation (25) in the proof of Theorem 1, we do not need the full inequalities (10) but only require that $\alpha\ge1$ satisfies the first inequality of Equation (10), that is,

\begin{equation*} r \gt \max\left\{\frac{(q-1)(\alpha+1)^2}{4\alpha} + 1, q+\frac{\alpha-1}{2}\right\}. \end{equation*}

Clearly, this inequality holds when α = 1. Moreover, if α = 1, then Equation (18) holds with m = 2. Hence, we can proceed as in the proof of Theorem 1 with α = 1, m = 2 and ϕ_R (instead of η_R) until we reach an estimate of type Equation (25). More precisely, by setting

\begin{align*} L_R(u) &:= \int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x) |\nabla u|^q\right) \phi_R^{2q}\\&\qquad + \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^r \phi_R^{q}(x) f(y)|u(y)|^r \phi_R^{q}(y)}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y, \end{align*}

instead of Equation (19), we have

\begin{equation*} L_R(u) \le C_0 + C \int_{B_{2R}\setminus B_R}|u|^p \phi_R^{2q-p} |\nabla\phi_R|^p + C \int_{B_{2R}\setminus B_R}w(x) |u|^q \phi_R^{q} |\nabla\phi_R|^q, \end{equation*}

where

\begin{equation*} C_0 := C \int_{B_{R_0+1}\setminus B_{R_0}}|u|^p \phi_R^{2q-p} |\nabla\phi_R|^p + C \int_{B_{R_0+1}\setminus B_{R_0}}w(x) |u|^q \phi_R^{q} |\nabla\phi_R|^q. \end{equation*}

Then proceeding as in the proof of Equation (25), we obtain

(31)\begin{equation} L_R(u) \le C_0 + C R^{N-p-\frac{p(N+\mu+2b)}{2r}} [L_R(u)]^\frac{p}{2r} + C R^{N+a-q-\frac{q(N+\mu+2b)}{2r}} [L_R(u)]^\frac{q}{2r}. \end{equation}

Suppose on the contrary that

(32)\begin{equation} \int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x)|\nabla u|^q\right) + \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r} f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y = \infty, \end{equation}

then there exists $R_1 \gt R_0+1$ such that

\begin{equation*} L_{R_1}(u) \gt 2C_0. \end{equation*}

Hence, for all $R \gt R_1$, estimate Equation (31) yields

\begin{equation*} L_R(u) \le 2C R^{N-p-\frac{p(N+\mu+2b)}{2r}} [L_R(u)]^\frac{p}{2r} + 2C R^{N+a-q-\frac{q(N+\mu+2b)}{2r}} [L_R(u)]^\frac{q}{2r}. \end{equation*}

Since $p\leq q$, then by Equation (32), we obtain

\begin{equation*} L_R(u) \le \left(2C R^{N-p-\frac{p(N+\mu+2b)}{2r}} + 2C R^{N+a-q-\frac{q(N+\mu+2b)}{2r}}\right) [L_R(u)]^\frac{q}{2r}. \end{equation*}

Hence,

(33)\begin{equation} [L_R(u)]^\frac{2r-q}{2r} \le 2C R^{N-p-\frac{p(N+\mu+2b)}{2r}} + 2C R^{N+a-q-\frac{q(N+\mu+2b)}{2r}}. \end{equation}

From Equation (30), we have $2r(N-p)\leq p(N+\mu+2b)$ and $2r(N+a-q)\leq q(N+\mu+2b)$. Note that r > q, by letting $R\to+\infty$ in Equation (33), we get a contradiction with Equation (32). This proves the lemma.

Lemma 5. (A Pohožaev-type identity)

Assume $2\le p \lt N$, $p \le q \lt N+a$, $\max\{0,N+2(a-q-b)\} \lt \mu \lt N$ and r > q. Let u be a finite energy solution of Equation (9) such that $|\nabla u|^{p-2} \nabla u + |x|^a|\nabla u|^{q-2} \nabla u \in W^{1,2}_{\rm loc}(\mathbb{R}^N,\mathbb{R}^N)$. Then,

\begin{align*} \frac{N-p}{p} \int_{\mathbb{R}^N}& |\nabla u|^p + \frac{N+a-q}{q} \int_{\mathbb{R}^N} |x|^a |\nabla u|^q\\&\qquad =\frac{N+\mu +2b}{2r} \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|x|^b |u(x)|^{r} |y|^b |u(y)|^{r}}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y. \end{align*}

Proof. By density arguments, we can use $v_R(x) = \eta_R(x) x \cdot \nabla u(x)$ as a test function in Equation (6), where η_R is defined as in Equation (17). Hence, we obtain

(34)\begin{equation} \begin{aligned} & \int_{\mathbb{R}^N} \left(|\nabla u|^{p-2} \nabla u \cdot \nabla v_R + |x|^a|\nabla u|^{q-2} \nabla u \cdot \nabla v_R\right)\\ &\qquad = \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|x|^b |u(x)|^{r-2} u(x) v_R(x) |y|^b |u(y)|^{r}} {|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y. \end{aligned} \end{equation}

We compute the limits of integrals in Equation (34) when $R\to+\infty$. First of all, we have

\begin{align*} &\int_{\mathbb{R}^N} |\nabla u|^{p-2} \nabla u \cdot \nabla v_R = \int_{\mathbb{R}^N} |\nabla u|^{p-2} \nabla u \cdot \nabla \left(\eta_R(x) x \cdot \nabla u\right)\\ &= \int_{\mathbb{R}^N} \eta_R(x) |\nabla u|^p + \int_{\mathbb{R}^N} \eta_R(x) x\cdot\nabla\left(\frac{|\nabla u|^p}{p}\right) + \int_{\mathbb{R}^N} |\nabla u|^{p-2} (\nabla u\cdot\nabla\eta_R) (x\cdot\nabla u)\\ &= - \int_{\mathbb{R}^N} \left[(N-p)\eta_R + x\cdot\nabla\eta_R\right] \frac{|\nabla u|^p}{p} + \int_{\mathbb{R}^N} |\nabla u|^{p-2} (\nabla u\cdot\nabla\eta_R) (x\cdot\nabla u). \end{align*}

Since $\int_{\mathbb{R}^N} |\nabla u|^p \lt \infty$, by the dominated convergence theorem, we derive

(35)\begin{equation} \lim_{R\to+\infty} \int_{\mathbb{R}^N} |\nabla u|^{p-2} \nabla u \cdot \nabla v_R = -\frac{N-p}{p}\int_{\mathbb{R}^N} |\nabla u|^p. \end{equation}

In the same way,

\begin{align*} &\int_{\mathbb{R}^N} |x|^a |\nabla u|^{q-2} \nabla u \cdot \nabla v_R = \int_{\mathbb{R}^N} |x|^a |\nabla u|^{q-2} \nabla u \cdot \nabla \left(\eta_R(x) x \cdot \nabla u\right)\\ &= \int_{\mathbb{R}^N} |x|^a \eta_R(x) |\nabla u|^q + \int_{\mathbb{R}^N} |x|^a \eta_R(x) x\cdot\nabla\left(\frac{|\nabla u|^q}{q}\right)\\&\qquad + \int_{\mathbb{R}^N} |x|^a |\nabla u|^{q-2} (\nabla u\cdot\nabla\eta_R) (x\cdot\nabla u)\\ &= - \int_{\mathbb{R}^N} \left[(N-q+a)|x|^a\eta_R + |x|^a x\cdot\nabla\eta_R\right] \frac{|\nabla u|^q}{q}\\&\qquad + \int_{\mathbb{R}^N} |x|^a |\nabla u|^{q-2} (\nabla u\cdot\nabla\eta_R) (x\cdot\nabla u). \end{align*}

Since $\int_{\mathbb{R}^N} |x|^a|\nabla u|^q \lt \infty$, the dominated convergence theorem gives us

(36)\begin{equation} \lim_{R\to+\infty} \int_{\mathbb{R}^N} |x|^a |\nabla u|^{q-2} \nabla u \cdot \nabla v_R = -\frac{N+a-q}{q}\int_{\mathbb{R}^N} |x|^a |\nabla u|^q. \end{equation}

Now we compute the right-hand side of Equation (34) as follows:

\begin{align*} &\int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|x|^b |u(x)|^{r-2} u(x) v_R(x) |y|^b |u(y)|^{r}}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\ &= \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|y|^b|u(y)|^r}{|x-y|^{N-\mu}} |x|^b\eta_R(x) x\cdot\nabla\left(\frac{|u(x)|^r}{r}\right) {\rm d}x\,{\rm d}y\\ &= -\int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|y|^b|u(y)|^r}{|x-y|^{N-\mu}} \left[(N+b)\eta_R(x) + x\cdot\nabla\eta_R(x)\right] \frac{|x|^b |u(x)|^r}{r} {\rm d}x\,{\rm d}y\\ &\qquad+\frac{N-\mu}{2} \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|y|^b|u(y)|^r}{|x-y|^{N-\mu}} \frac{(x-y)\cdot(x\eta_R(x)-y\eta_R(y))}{|x-y|^2} \frac{|x|^b|u(x)|^r}{r} {\rm d}x\,{\rm d}y. \end{align*}

Since $\int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|x|^b |u(x)|^{r} |y|^b |u(y)|^{r}}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \lt \infty$, we use the dominated convergence theorem again to deduce that

(37)\begin{equation} \begin{split} \lim_{R\to+\infty} \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|x|^b |u(x)|^{r-2} u(x) v_R(x) |y|^b |u(y)|^{r}}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\\= -\frac{N+\mu +2b}{2r} \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{|x|^b |u(x)|^{r} |y|^b |u(y)|^{r}}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y. \end{split} \end{equation}

By collecting Equation (34)–(37), we obtain the desired identity.

Proof of Theorem 2

Using $\varphi=u\eta^2_R$ as a test function in Equation (6) with $w(x)=|x|^a$ and $f(x)=|x|^b$, where η_R is chosen as in the proof of Theorem 1, we obtain

(38)\begin{equation} \begin{split} &\int_{\mathbb{R}^N} |\nabla u|^p \eta^2_R + \int_{\mathbb{R}^N} |\nabla u|^{p-2} u \nabla u \cdot \nabla\eta^2_R \\ &\qquad +\int_{\mathbb{R}^N} w(x) |\nabla u|^q \eta^2_R + \int_{\mathbb{R}^N} w(x) |\nabla u|^{q-2} u \nabla u \cdot \nabla\eta^2_R \\ & \qquad \qquad \qquad = \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r} \eta^2_R(x) f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y. \end{split} \end{equation}

Note that

\begin{align*} \left(\int_{\mathbb{R}^N} f(x) \eta^r_R|u|^{r} \right)^2 &= \int_{\mathbb{R}^N} \int_{\mathbb{R}^N} f(x) |u(x)|^{r} \eta_R(x)^r f(y) |u(y)|^{r} \eta_R(y)^r {\rm d}x\,{\rm d}y \\ & \leq R^{N-\mu}\int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r} \eta_R(x)^r f(y)|u(y)|^r \eta_R(y)^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y \end{align*}

and

\begin{align*} &\left| \int_{\mathbb{R}^N} |\nabla u|^{p-2} u \nabla u \cdot \nabla\eta^2_R \right| \le 2\int_{\mathbb{R}^N} |\nabla u|^{p-1} |u| \eta_R|\nabla\eta_R| \\ &\le 2 \left(\int_{\mathbb{R}^N} |\nabla u|^p \right)^\frac{p-1}{p} \left(\int_{\mathbb{R}^N} f(x) \eta^r_R|u|^{r} \right)^\frac{1}{r} \left(\int_{\mathbb{R}^N} f(x)^{-\frac{p}{r-p}} |\nabla\eta_R|^\frac{pr}{r-p} \right)^\frac{r-p}{pr}\\ &\le 2R^{\frac{N-\mu}{2r}} \left(\int_{\mathbb{R}^N} f(x)^{-\frac{p}{r-p}} |\nabla\eta_R|^\frac{pr}{r-p} \right)^\frac{r-p}{pr} \\ & \qquad \times \left(\int_{\mathbb{R}^N} |\nabla u|^p \right)^\frac{p-1}{p}\left(\int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r} \eta_R(x)^r f(y)|u(y)|^r \eta_R(y)^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\right)^{\frac{1}{2r}} \\ &\le C R^{\frac{N-\mu}{2r}+\frac{N(r-p)}{pr} - \frac{r+b}{r}} \left(\int_{\mathbb{R}^N} |\nabla u|^p \right)^\frac{p-1}{p} \left(\int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r} f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\right)^{\frac{1}{2r}}. \end{align*}

If $\frac{N+\mu +2b}{2r} \gt \frac{N-p}{p}$, then $\frac{N-\mu}{2r}+\frac{N(r-p)}{pr} - \frac{r+b}{r} \lt 0$. Therefore, by letting $R \to \infty$, we gather that

(39)\begin{equation} \lim\limits_{R\to+\infty} \int_{\mathbb{R}^N} |\nabla u|^{p-2} u \nabla u \cdot \nabla\eta^2_R = 0. \end{equation}

Similarly,

\begin{align*} &\left| \int_{\mathbb{R}^N} w(x) |\nabla u|^{q-2} u \nabla u \cdot \nabla\eta^2_R \right| \le 2 \int_{\mathbb{R}^N} w(x) |\nabla u|^{q-1} |u| \eta_R |\nabla\eta_R| \\ &\le 2 \left(\int_{\mathbb{R}^N} w(x) |\nabla u|^q \right)^\frac{q-1}{q} \left(\int_{\mathbb{R}^N} f(x) \eta^r_R|u|^{r} \right)^\frac{1}{r} \left(\int_{\mathbb{R}^N} w(x)^\frac{r}{r-q} f(x)^{-\frac{q}{r-q}} |\nabla\eta_R|^\frac{qr}{r-q} \right)^\frac{r-q}{qr}\\ &\le C R^{\frac{N-\mu}{2r}+\frac{N(r-q)}{qr} +\frac{a}{q} - \frac{r+b}{r}} \left(\int_{\mathbb{R}^N} w(x) |\nabla u|^q \right)^\frac{q-1}{q}\\&\qquad \left(\int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r} f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y\right)^{\frac{1}{2r}}. \end{align*}

If $\frac{N+\mu +2b}{2r} \gt \frac{N+a-q}{q}$, we have $\frac{N-\mu}{2r}+\frac{N(r-q)}{qr} +\frac{a}{q} - \frac{r+b}{r} \lt 0$. Therefore, by letting $R \to \infty$, we gather that

(40)\begin{equation} \lim\limits_{R\to+\infty} \int_{\mathbb{R}^N} w(x) |\nabla u|^{q-2} u \nabla u \cdot \nabla\eta^2_R = 0. \end{equation}

As a consequence, it follows from Equations (38)–(40) that if $\frac{N+\mu +2b}{2r} \gt \max\left\{\frac{N-p}{p}, \frac{N+a-q}{q}\right\}$, then

(41)\begin{equation} \int_{\mathbb{R}^N} \left(|\nabla u|^p + w(x)|\nabla u|^q\right) = \int_{\mathbb{R}^N}\int_{\mathbb{R}^N} \frac{f(x)|u(x)|^{r} f(y)|u(y)|^r}{|x-y|^{N-\mu}} {\rm d}x\,{\rm d}y. \end{equation}

If $\frac{N+\mu +2 b}{2r} = \max\left\{\frac{N-p}{p}, \frac{N+a-q}{q}\right\}$, then the conclusion follows from Lemmas 4 and 5.

If $\frac{N+\mu +2b}{2r} \gt \max\left\{\frac{N-p}{p}, \frac{N+a-q}{q}\right\}$, then we may exploit Lemmas 4 and 5 and Equation (41) to obtain

\begin{equation*} \left(\frac{N+\mu +2b}{2r} - \frac{N-p}{p}\right) \int_{\mathbb{R}^N} |\nabla u|^p + \left(\frac{N+\mu +2b}{2r} - \frac{N+a-q}{q}\right) \int_{\mathbb{R}^N} |x|^a |\nabla u|^q = 0. \end{equation*}

This implies that u is constant and hence must be zero.