Hardy–Littlewood–Sobolev inequality and existence of the extremal functions with extended kernel

Abstract

In this paper, we consider the following Hardy–Littlewood–Sobolev inequality with extended kernel(0.1)

\begin{equation} \int_{\mathbb{R}_+^{n}}\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta}}{|x-y|^{n-\alpha}}f(y)g(x) {\rm d}y{\rm d}x\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}_+^{n})} \|g\|_{L^{q'}(\mathbb{R}_+^{n})}, \end{equation}

for any nonnegative functions $f\in L^{p}(\partial \mathbb {R}_+^{n})$, $g\in L^{q'}(\mathbb {R}_+^{n})$ and $p,\,\ q'\in (1,\,\infty )$, $\beta \geq 0$, $\alpha +\beta >1$ such that $\frac {n-1}{n}\frac {1}{p}+\frac {1}{q'}-\frac {\alpha +\beta -1}{n}=1$.

We prove the existence of all extremal functions for (0.1). We show that if $f$ and $g$ are extremal functions for (0.1) then both of $f$ and $g$ are radially decreasing. Moreover, we apply the regularity lifting method to obtain the smoothness of extremal functions. Finally, we derive the sufficient and necessary condition of the existence of any nonnegative nontrivial solutions for the Euler–Lagrange equations by using Pohozaev identity.

1. Introduction

The classical Hardy–Littlewood–Sobolev inequality that was obtained by Hardy and Littlewood [36] for $n = 1$ and by Sobolev [50] for general $n$ states that

(1.1)\begin{equation} \int_{\mathbb{R}^{n}}\int_{\mathbb{R}^{n}}|x-y|^{-(n-\alpha)} f(x)g(y){\rm d}x{\rm d}y\leq C_{\alpha,n,p}\|f\|_{L^{p}(\mathbb{R}^{n})}\|g\|_{L^{q'}(\mathbb{R}^{n})} \end{equation}

with $1< p,\,q'<\infty,\, 0<\alpha < n$ and $\frac {1}{p}+\frac {1}{q'}+\frac {n-\alpha }{n}=2$.

Lieb [39] employed the rearrangement inequalities to obtain the existence of the extremal functions of inequality (1.1). Furthermore, they also classified extremals of the inequality (1.1) and computed the sharp constant $C_{\alpha,n,p}$ only when one of $p$ and $q'$ is equal to $2$ or $p=q'$.

Through the inequality (1.1), we can deduce many important geometrical inequalities such as the Gross logarithmic Sobolev inequality [31] and the Moser–Onofri–Beckner inequality [1]. It is also well-known that if we pick $\alpha =2,\, p = q' = {2n}/{(n+2)}$, then the Hardy–Littlewood– Sobolev inequality is in fact equivalent to the Sobolev inequality by Green's representation formula. By using the competing symmetry method, Carlen and Loss [8] provided a different proof from Lieb's of the sharp constants and extremal functions in the diagonal case $p = q' = {2n}/{(n+\alpha )}$ and Frank and Lieb [25] offered a new proof using the reflection positivity of inversions in spheres in the special diagonal case. Frank and Lieb [26] further employed a rearrangement-free technique developed in [27] to recapture the best constant of inequality (1.1). Folland and Stein [24] extended the inequality (1.1) to the Heisenberg group and established the Hardy–Littlewood–Sobolev inequality on Heisenberg group. Frank and Lieb [27] classify the extremals of this inequality in the diagonal case. This extends the earlier work of Jerison and Lee [38] for sharp constants and extremals for the Sobolev inequality on the Heisenberg group in the conformal case in their study of CR Yamabe problem. Furthermore, Han et al. [34] established the double-weighted Hardy–Littlewood–Sobolev inequality (namely, Stein–Weiss inequality) on the Heisenberg group and discussed the regularity and asymptotic behaviour of the extremal functions. Recently, Chen et al. [13] used the concentration-compactness principle to obtain existence of extremals of the Stein–Weiss inequality on the Heisenberg group for all indices. We also mention that when $p = q' = {2n}/{(n+\alpha )}$, Euler–Lagrange equation of the extremals to the Hardy–Littlewood–Sobolev inequality in the Euclidean space is a conformal invariant integral equation. The inequality (1.1) and its extensions have many applications in partial differential equations. Some remarkable extensions have already been obtained on the upper half space by Dou and Zhu [22], on compact Riemannian manifolds by Han and Zhu [35] and the reversed (weighted) Hardy–Littlewood–Sobolev inequality in [10, 23, 48, 49]. For more results about the (weighted) Hardy–Littlewood–Sobolev inequality, the general weighted inequalities and their corresponding Euler–Lagrange equations, refer to e.g. [2, 3, 9, 15–20, 28, 32, 37, 42–45, 47, 51] and the references therein.

Recently, Gluck [30] proved the following sharp Hardy–Littlewood–Sobolev inequality with extended kernel in the conformal invariant case ($p=\tfrac {2(n-1)}{n+\alpha -2}$, $q'=\tfrac {2n}{n+\alpha +2\beta }$)

(1.2)\begin{equation} \Big|\int_{\mathbb{R}_+^{n}}\int_{\partial\mathbb{R}^{n}_+} K(x'-y,x_n) f(y)g(x) {\rm d}y{\rm d}x\Big|\leq C_{n,\alpha,\beta,p} \|f\|_{L^{p}(\partial\mathbb{R}_+^{n})} \|g\|_{L^{q'}(\mathbb{R}_+^{n})}. \end{equation}

where $K$ is a kernel of the form

\[ K(x',x_n)=K_{\alpha,\beta}(x',x_n)=\frac{x_n^{\beta}}{(|x'|^{2}+x_n^{2})^{{(n-\alpha)}/{2}}}, \quad x=(x',x_n)\in \mathbb{R}^{n-1}\times (0,\infty), \]

and $\alpha$, $\beta$ satisfy $\beta \geq 0$, $0<\alpha +\beta < n-\beta$,

(1.3)\begin{equation} \frac{n-\alpha-2\beta}{2n}+\frac{n-\alpha}{2(n-1)}<1. \end{equation}

In fact, for $\alpha =0$, $\beta =1$, the kernel $K_{\alpha,\beta }$ is the classical Poisson kernel. Hang et al. [33] derived the Hardy–Littlewood–Sobolev inequality with the Poisson kernel and proved the existence of extremals for this inequality by the concentration-compactness principle [40, 41]. For the conformal invariant case, they classified the extremal functions of the inequality, and computed the sharp constant. Integral inequality with the Poisson kernel is highly related to Carleman's proof of isoperimetric inequality in the plane (see [7]). For $\alpha \in (0,\,1)$, $\beta =1-\alpha$, the kernel $K_{\alpha,\beta }$ is related to the divergence form operator $u\mapsto {\rm div}(x_n^{\alpha } \nabla u)$ (the poly-harmonic extension operator) on the half space. Chen [14] established sharp Hardy–Littlewood–Sobolev inequality (1.2). He also generalized Carleman's inequality for harmonic functions in the plane to poly-harmonic functions in higher dimensions. Dou and Zhu [22] studied the sharp Hardy–Littlewood–Sobolev inequality on the upper half space and the existences of extremal functions for $\beta =0$. Dou et al. [21] investigated the integral inequality (1.2) in the special index through the methods based on conformal transformation for $\beta =1$. Different from Dou et al. [21], Chen et al. [12] derived the Hardy–Littlewood–Sobolev inequality to all critical index for $\beta =1$. Furthermore, Chen et al. [11] extended it to the weighted Hardy–Littlewood–Sobolev inequality.

In this paper, we extended the Hardy–Littlewood–Sobolev inequality with extended kernel in the conformal invariant case to all critical index. That is,

Theorem 1.1 Let $n\geq 2,$ $1< p,\, q'<\infty,$ $\beta \geq 0,$ $\alpha +\beta >1$ and suppose that $\alpha,$ $\beta,$ $p,$ $q'$ satisfy

\[ \frac{n-1}{n}\frac{1}{p}+\frac{1}{q'}-\frac{\alpha+\beta-1}{n}=1. \]

Then there is a constant $C_{n,\alpha,\beta,p}>0$ such that for any nonnegative functions $f\in L^{p}(\partial \mathbb {R}_+^{n}),$ $g\in L^{q'}(\mathbb {R}_+^{n}),$

(1.4)\begin{equation} \int_{\mathbb{R}_+^{n}}\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n-\alpha}} f(y)g(x) {\rm d}y{\rm d}x\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}_+^{n})} \|g\|_{L^{q'}(\mathbb{R}_+^{n})}. \end{equation}

We remark that the constant $C_{n,\alpha,\beta,p}$ above can be considered as the least one such that the above inequality holds for all nonnegative functions $f\in L^{p}(\partial \mathbb {R}^{n}_+)$, $g\in L^{q'}(\mathbb {R}^{n}_+)$. This constant $C_{n,\alpha,\beta,p}$ is often referred as the best constant for the Hardy–Littlewood–Sobolev inequality with extended kernel.

Define

\[ Tf(x)=\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{|x-y|^{n-\alpha}}f(y){\rm d}y,\quad T'g(y)=\int_{\partial\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{|x-y|^{n-\alpha}}g(x){\rm d}x. \]

Throughout this paper, we always assume that $q$ and $q'$ are conjugate numbers. That is, $q$ and $q'$ satisfy $\frac {1}{q}+\frac {1}{q'}=1$. By duality, it is easy to verify that the inequality (1.4) is equivalent to the following two corollaries.

Corollary 1.2 Assume that $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta >1,$ $1< p<\frac {n-1}{\alpha +\beta -1},$ and

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{p}-\frac{\alpha+\beta-1}{n-1}\right). \]

Then there is a constant $C_{n,\alpha,\beta,p}>0$ such that

(1.5)\begin{equation} \|Tf\|_{L^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}. \end{equation}

Corollary 1.3 Assume that $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta >1,$ $1< q'<\frac {n}{\alpha +\beta },$ and

\[ \frac{1}{p'}=\frac{n}{n-1}\left(\frac{1}{q'}-\frac{\alpha+\beta}{n}\right). \]

Then there is a constant $C_{n,\alpha,\beta,q'}>0$ such that

(1.6)\begin{equation} \|T'g\|_{L^{p'}(\partial\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|g\|_{L^{q'}(\mathbb{R}^{n}_+)}. \end{equation}

Once we establish the Hardy–Littlewood–Sobolev inequality with extended kernel, it is natural to ask whether the extremal functions for inequality (1.4) actually exist. To answer this question, we turn to consider the following maximizing problem

(1.7)\begin{equation} C_{n,\alpha,\beta,p}:= \sup \{\|Tf\|_{L^{q}(\mathbb{R}^{n}_+)} \mid \|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=1, f\geq 0\}, \end{equation}

where $p$, $q$ satisfy

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{p}-\frac{\alpha+\beta-1}{n-1}\right). \]

It is not hard to verify that the extremals of inequality (1.5) are those solving the maximizing problem (1.7). We use the rearrangement inequality to prove the attainability of maximizers for the maximizing problem (1.7).

Theorem 1.4 Let $n\geq 2,$ $1< p,\, q<\infty,$ $\beta \geq 0,$ $\alpha +\beta >1,$ and suppose that $\alpha,$ $\beta,$ $p,$ $q$ satisfy

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{p}-\frac{\alpha+\beta-1}{n-1}\right). \]

Then there exists some function $f\in L^{p}(\partial \mathbb {R}^{n}_+)$ such that $f\geq 0,$ $\|f\|_{L^{p}(\partial \mathbb {R}^{n}_+)}=1,$ and $\|Tf\|_{L^{q}( \mathbb {R}^{n}_+)}=C_{n,\alpha,\beta,p}$. Moreover, all extremal functions are radially symmetric and strictly decreasing about some point $y_0\in \partial \mathbb {R}^{n}_+$.

We now turn our attention to study the regularity of the extremal functions for inequality (1.5), the Euler–Lagrange equation for extremal functions, up to a constant multiplier, is given by

(1.8)\begin{equation} f^{p-1}(y)=\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{|x-y|^{n-\alpha}}(Tf(x))^{q-1}{\rm d}x. \end{equation}

We prove

Theorem 1.5 Let $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta > 1$ and $1 < p< \frac {n-1}{\alpha +\beta -1}$. Suppose that $f\in L_{loc}^{p}(\partial \mathbb {R}^{n}_+)$ is nonnegative solution to (1.8) with $\frac {1}{q}=\frac {n-1}{n}(\frac {1}{p}-\frac {\alpha +\beta -1}{n-1})$. Then $f\in C^{\infty }(\partial \mathbb {R}^{n}_+)$.

Assume that

\[ u(y)=f^{p-1}(y), \quad v(x)=Tf(x). \]

Denote

\[ \theta=\frac{1}{p-1},\quad \kappa=q-1. \]

Euler–Lagrange equation (1.8) can be rewritten as the following integral system

(1.9)\begin{equation} \begin{cases} u(y)=\displaystyle\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}} v^{\kappa}(x){\rm d}x, & y\in\partial\mathbb{R}^{n}_+,\\ v(x)=\displaystyle\int_{\partial\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}}u^{\theta}(y) {\rm d}y, & x\in\mathbb{R}^{n}_+. \end{cases} \end{equation}

We use the Pohozaev identity to prove the following theorem.

Theorem 1.6 For $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta > 1,$ $\theta >0,$ $\kappa >0,$ assume that $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ is a pair of nonnegative nontrivial $C^{1}$ solutions of (1.9), then a necessary condition for $\theta$ and $\kappa$ is

\[ \frac{n-1}{\theta+1}+\frac{n}{\kappa+1}=n-\alpha-\beta. \]

Obviously, extremals $(f,\,g)$ of inequality (1.4) satisfies the integral system (1.9). In light of theorems 3.1, 4.1 and 5.1, we obtain the sufficient and necessary condition for existence of positive solutions to the integral system (1.9).

Theorem 1.7 For $\theta >0,$ $\kappa >0,$ let $n,$ $\alpha,$ $\beta,$ $p,$ $q$ satisfy all the hypotheses of theorems 3.1, 4.1 and 5.1, then the sufficient and necessary condition for the existence of a pair of nonnegative nontrivial solutions $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ to system (1.9) is

\[ \frac{n-1}{\theta+1}+\frac{n}{\kappa+1}=n-\alpha-\beta. \]

The following Liouville type theorem was proved by Gluck.

Theorem 1.8 (see [30])

Let $n\geq 2$ and suppose $\alpha,$ $\beta$ satisfy $\beta \geq 0,$ $0<\alpha +\beta < n-\beta$ and (1.3). If $u\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)$ and $v\in L^{\kappa +1}(\mathbb {R}^{n}_+)$ are positive solutions of (1.9) with $\theta =\frac {n+\alpha -2}{n-\alpha }$ and $\kappa =\frac {n+\alpha +2\beta }{n-\alpha -2\beta }$. Then there exists $c_1>0,$ $d>0$ and $y_0\in \partial \mathbb {R}^{n}_+$ such that

\[ u(y)=\frac{c_1}{\big(d^{2}+|y-y_0|^{2}\big)^{{(n-\alpha)}/{2}}} \ {\rm for\ all\ } \ y\in \partial\mathbb{R}^{n}_+. \]

With the help of theorem 1.7, we use weaker assumption (1.10) to obtain theorem 1.9 instead of the conformal invariant case.

Theorem 1.9 Let $n\geq 2$ and suppose $\alpha,$ $\beta$ satisfy $\beta \geq 0,$ $0<\alpha +\beta < n-\beta$. If $u\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)$ and $v\in L^{\kappa +1}(\mathbb {R}^{n}_+)$ are nonnegative nontrivial solutions of (1.9) with

(1.10)\begin{equation} 0<\theta\leq\frac{n+\alpha-2}{n-\alpha},\quad 0<\kappa\leq\frac{n+\alpha+2\beta}{n-\alpha-2\beta}. \end{equation}

Then

\[ \theta=\frac{n+\alpha-2}{n-\alpha},\quad \kappa=\frac{n+\alpha+2\beta}{n-\alpha-2\beta}. \]

Moreover, there exists $c_1>0$, $d>0$ and $y_0\in \partial \mathbb {R}^{n}_+$ such that

\[ u(y)=\frac{c_1}{\big(d^{2}+|y-y_0|^{2}\big)^{{(n-\alpha)}/{2}}} \ {\rm for\ all\ } \ y\in \partial\mathbb{R}^{n}_+. \]

From theorem 1.7, we must have $\theta =\frac {n+\alpha -2}{n-\alpha }$ and $\kappa =\frac {n+\alpha +2\beta }{n-\alpha -2\beta }$. Then, the proof is completely similar to the proof by Gluck in [30], so we omit the details.

This paper is organized as follows. In § 2, we prove the Hardy–Littlewood–Sobolev inequality with the extended kernel. In § 3, by the rearrangement inequality, we obtain the existence of extremals of the inequality. Section 4 is devoted to the regularity estimate of the extremal functions of the Hardy–Littlewood–Sobolev inequality with the extended kernel. In § 5, using the Pohozaev identity in integral forms, we give sufficient and necessary conditions for the existence of nonnegative nontrivial solutions.

2. The proof of theorem 2.1

In this section, we use the Marcinkiewicz interpolation theorem and weak type estimate to establish the Hardy–Littlewood–Sobolev inequality with the extended kernel.

Theorem 2.1 Let $n\geq 2,$ $1< p,\, q'<\infty,$ $\beta \geq 0,$ $\alpha +\beta >1$ and suppose that $\alpha,$ $\beta,$ $p,$ $q'$ satisfy

\[ \frac{n-1}{n}\frac{1}{p}+\frac{1}{q'}-\frac{\alpha+\beta-1}{n}=1. \]

Then there is a constant $C_{n,\alpha,\beta,p}>0$ such that for any nonnegative functions $f\in L^{p}(\partial \mathbb {R}_+^{n})$, $g\in L^{q'}(\mathbb {R}_+^{n}),$

(2.1)\begin{equation} \int_{\mathbb{R}_+^{n}}\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n-\alpha}} f(y)g(x) {\rm d}y{\rm d}x\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}_+^{n})} \|g\|_{L^{q'}(\mathbb{R}_+^{n})}. \end{equation}

Proof. For $t>0$ and $x'\in \mathbb {R}^{n-1}$, define

\[ K_t(x')=\frac{t^{\beta}}{(|x'|^{2}+t^{2})^{{(n-\alpha)}/{2}}}. \]

Then, for $x=(x',\,x_n)\in \mathbb {R}^{n}_+$, $y\in \partial \mathbb {R}^{n}_+$, we have

\[ K(x'-y,x_n)=K_{x_n}(x'-y),\quad Tf(x)=(K_{x_n}*f)(x'). \]

We are ready to prove theorem 2.1 via proving inequality (1.5). For $p\in (1,\,\frac {n-1}{\alpha +\beta -1})$ and $q$ given by $\frac {1}{q}=\frac {n-1}{n}(\frac {1}{p}-\frac {\alpha +\beta -1}{n-1})$. By the Marcinkiewicz interpolation theorem (see [52]), we only need to prove the following weak-type estimate:

(2.2)\begin{equation} \|Tf\|_{L_w^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}. \end{equation}

That is, we need to show that there is a constant $C_{n,\alpha,\beta,p}>0$ such that

\[ \lambda|\{x\in\mathbb{R}^{n}_+{\mid} |Tf(x)|>\lambda\}|^{{1}/{q}}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}, \quad \forall f\in L^{p}(\partial\mathbb{R}^{n}_+), \ \forall \lambda>0. \]

Without the loss of generality, we may assume that $\|f\|_{L^{p}(\partial \mathbb {R}^{n}_+)}=1$. Assume that $r,\,s$ satisfy

(2.3)\begin{equation} r\in \left(\frac{(n-1)p}{(1-\alpha)p+n-1}, \frac{np}{(1-\alpha-\beta)p+n-1}\right),\quad \frac{1}{r}+1=\frac{1}{p}+\frac{1}{s},\ s\geq 1. \end{equation}

It follows from the Young equality that

\begin{align*} & \int_{\substack{x\in\mathbb{R}^{n}_+ \\ 0< x_n< a}}|Tf(x)|^{r}{\rm d}x\\ & \quad=\int_0^{a}\int_{\mathbb{R}^{n-1}}|(K_{x_n}*f)(x')|^{r}{\rm d}x'{\rm d}x_n\\ & \quad\leq\|f\|_{L^{p}(\mathbb{R}^{n-1})}\int_0^{a}\|K_{x_n}\|^{r}_{L^{s}(\mathbb{R}^{n-1})}{\rm d}x_n\\ & \quad=\int_0^{a}\left(\int_{\mathbb{R}^{n-1}}\frac{x_n^{\beta s}}{(|x'|^{2}+x_n^{2})^{{((n-\alpha)s)}/{2}}}{\rm d}x'\right)^{{r}/{s}}{\rm d}x_n\\ & \quad\leq\int_0^{a}x_n^{{((n-1)r)}/{s}+(\alpha+\beta-n)r}{\rm d}x_n\left(\int_{\mathbb{R}^{n-1}}\frac{1}{(|x'|^{2}+1)^{{((n-\alpha)s)}/{2}}}{\rm d}x'\right)^{{r}/{s}}. \end{align*}

One can deduce from (2.3) that

\[ \frac{(n-1)r}{s}+(\alpha+\beta-n)r>{-}1,\quad (n-\alpha)s>n-1. \]

Then, we have

\[ \int_{\substack{x\in\mathbb{R}^{n}_+ \\ 0< x_n< a}}|Tf(x)|^{r}{\rm d}x\leq C_1 a^{{((n-1)r)}/{s}+(\alpha+\beta-n)r+1}. \]

In view of the Hölder inequality and the integration of the extended kernel, we can see that

\[ \|K_{x_n}*f(x')\|_{L^{\infty}(\mathbb{R}^{n-1})}\leq C x_n^{{(n-1)}/{p'}+(\alpha+\beta-n)}. \]

Since $p\in (1,\,\frac {n-1}{\alpha +\beta -1})$, we know that $\frac {n-1}{p'}+(\alpha +\beta -n)<0$. Then, we derive that

\begin{align*} & |\{x\in\mathbb{R}^{n}_+{\mid} |Tf(x)|>\lambda\}|\\ & \quad=\Big|\Big\{x\in\mathbb{R}^{n}_+{\mid} 0< x_n< C \lambda^{{p'}/{(n-1+p'(\alpha+\beta-n))}},\quad |Tf(x)|>\lambda\Big\}\Big|\\ & \quad\leq \frac{1}{\lambda^{r}}\int_{x\in\mathbb{R}^{n}_+,\quad 0< x_n< C \lambda^{{p'}/{(n-1+p'(\alpha+\beta-n))}}}|Tf(x)|^{r}{\rm d}x\\ & \quad\leq C' \lambda^{{np}/{((\alpha+\beta-1)p-n+1)}}\\ & \quad\leq C' \lambda^{{-}q}, \end{align*}

which implies that

(2.4)\begin{equation} \|Tf\|_{L_w^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}. \end{equation}

Note that inequality (2.4) implies, via the Marcinkiewicz interpolation [52], that

\[ \|Tf\|_{L^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}. \]

or even slight stronger inequality

(2.5)\begin{equation} \|Tf\|_{L^{q}(\mathbb{R}^{n}_+)}\leq C_{n,\alpha,\beta,p}\|f\|_{L^{p,q}(\partial\mathbb{R}^{n}_+)}. \end{equation}

where Lorentz norm $\|\cdot \|_{L^{p,q}}$ is defined by

\[ \|u\|_{L^{p,q}}=p^{{1}/{q}}\left(\int_0^{\infty} t^{q} \mid |u|>t|^{{q}/{p}}\frac{{\rm d}t}{t}\right)^{{1}/{q}}. \]

3. The proof of theorem 3.1

In the following, we will employ rearrangement inequality to investigate the existence of maximizers for the maximizing problem

(3.1)\begin{equation} C_{n,\alpha,\beta,p}:= \sup \{\|Tf\|_{L^{q}(\mathbb{R}^{n}_+)} \mid \|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=1, f\geq 0\}. \end{equation}

We prove

Theorem 3.1 Let $n\geq 2,$ $1< p,\, q<\infty,$ $\beta \geq 0,$ $\alpha +\beta >1$ and suppose that $\alpha,$ $\beta,$ $p,$ $q$ satisfy

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{p}-\frac{\alpha+\beta-1}{n-1}\right). \]

Then there exists some function $f\in L^{p}(\partial \mathbb {R}^{n}_+)$ such that $f\geq 0,$ $\|f\|_{L^{p}(\partial \mathbb {R}^{n}_+)}=1,$ and $\|Tf\|_{L^{q}( \mathbb {R}^{n}_+)}=C_{n,\alpha,\beta,p}$. Moreover, all extremal functions are radially symmetric and strictly decreasing about some point $y_0\in \partial \mathbb {R}^{n}_+$.

Proof. Using symmetrization argument, we first show that the supremum of (3.1) is attained by radially symmetric functions. Now, we recall the important Riesz rearrangement inequality. Let $u$ be a measurable function on $\mathbb {R}^{n}$, the symmetric rearrangement of $u$ is the nonnegative lower semi-continuous radial decreasing function $u^{*}$ that has the same distribution as $u$. Then, we have

\[ \int_{\mathbb{R}^{n}}{\rm d}x\int_{\mathbb{R}^{n}}u(x)v(y-x)w(y){\rm d}y\leq \int_{\mathbb{R}^{n}}{\rm d}x\int_{\mathbb{R}^{n}}u^{*}(x)v^{*}(y-x)w^{*}(y){\rm d}y. \]

Using the fact $\|w\|_{L^{p}(\mathbb {R}^{n})}=\|w^{*}\|_{L^{p}(\mathbb {R}^{n})}$ for $p>0$ and the standard duality argument, we see, for $1\leq p\leq \infty$,

\[ \|u*v\|_{L^{p}(\mathbb{R}^{n})}\leq\|u^{*}*v^{*}\|_{L^{p}(\mathbb{R}^{n})}. \]

Moreover, if $u$ is nonnegative radially symmetric and strictly decreasing in the radial direction, $v$ is nonnegative, $1< p<\infty$ and

\[ \|u*v\|_{L^{p}(\mathbb{R}^{n})}=\|u^{*}*v^{*}\|_{L^{p}(\mathbb{R}^{n})}<\infty, \]

then from Brascamp et al. [4], we have,

(3.2)\begin{equation} v(x)=v^{*}(x-x_0), \end{equation}

for some $x_0\in \mathbb {R}^{n}$.

Now, assume $f_i$ is a maximizing sequence in (3.1). Since

\[ \|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=\|f^{*}\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=1 \]

and

\begin{align*} \|Tf_i\|_{L^{q}(\mathbb{R}^{n}_+)}^{q}& =\int_0^{\infty}\|K_{x_n}*f_i\|_{L^{q}(\mathbb{R}^{n-1})}^{q}{\rm d}x_n\\ & \leq\int_0^{\infty}\|K_{x_n}*f_i^{*}\|_{L^{q}(\mathbb{R}^{n-1})}^{q}{\rm d}x_n\\ & =\|Tf_i^{*}\|_{L^{q}(\mathbb{R}^{n}_+)}^{q}. \end{align*}

We know that $f_i^{*}$ is also a maximizing sequence. Hence, we may assume $f_i$ is a nonnegative radial decreasing function.

For any $f\in L^{p}(\partial \mathbb {R}^{n}_+)$ and any $\lambda >0$, we let $f^{\lambda }(y)=\lambda ^{-({(n-1)}/{p})}f(\frac {y}{\lambda })$, then it is easy to check that

\[ \|f^{\lambda}\|_{L^{p}(\partial\mathbb{R}^{n}_+)}=\|f\|_{L^{p}(\partial\mathbb{R}^{n}_+)},\quad \|Tf^{\lambda}\|_{L^{q}(\mathbb{R}^{n}_+)}=\|Tf\|_{L^{q}(\mathbb{R}^{n}_+)}. \]

For convenience, denote $e_1^{'}=(1,\,0,\,\ldots,\,0)\in \mathbb {R}^{n-1}$ and

\[ a_i=\sup_{\lambda>0}f_i^{\lambda}(e_1^{'})=\sup_{\lambda>0}\lambda^{-({(n-1)}/{p})}f_i\left(\frac{e_1^{'}}{\lambda}\right). \]

It follows that

\[ 0\leq f_i(y)\leq a_i|y|^{{-({(n-1)}/{p})}} \]

and hence

\[ \|f_i\|_{L^{p,\infty}(\partial\mathbb{R}^{n}_+)}\leq \omega_{n-1}^{{1}/{p}}a_i. \]

Thus, by (2.5), we have

\begin{align*} \|Tf_i\|_{L^{q}(\mathbb{R}^{n}_+)}& \leq C_{n,\alpha,\beta,p}\|f_i\|_{L^{p,q}(\partial\mathbb{R}^{n}_+)}\\ & \leq C_{n,\alpha,\beta,p}\|f_i\|^{1-{p}/{q}}_{L^{p,\infty}(\partial\mathbb{R}^{n}_+)} \|f_i\|^{{p}/{q}}_{L^{p}(\partial\mathbb{R}^{n}_+)}\\ & \leq C_{n,\alpha,\beta,p}a_i^{1-{p}/{q}}, \end{align*}

which implies $a_i\geq c(n,\,\alpha,\,\beta,\, p)>0$. We may choose $\lambda _i>0$ such that $f_i^{\lambda _i}(e_1^{'})\geq c(n,\,\alpha,\,\beta,\, p)>0$. Replacing $f_i$ by $f_i^{\lambda _i}$, we may assume $f_i(e_1^{'})\geq c(n,\,\alpha,\,\beta,\, p)>0$. On the other hand, since $f_i$ is nonnegative radially decreasing and $f_i\in L^{p}(\partial \mathbb {R}^{n}_+)=1$, it is obvious that

\[ f_i(y)\leq \omega_{n-1}^{{1}/{p}}|y|^{{-({(n-1)}/{p})}}. \]

Hence after passing to a subsequence, we may find a nonnegative radially decreasing function $f$ such that $f_i\rightarrow f$ a.e. It follows that $f(y)\geq c(n,\,\alpha,\,\beta,\, p) > 0$ for $|y|\leq 1$, and $\|f\|_{ L^{p}(\partial \mathbb {R}^{n}_+)}\leq 1$. From Brezis and Lieb's Lemma [6], we see

\[ \int_{\partial\mathbb{R}^{n}_+}\big||f_i(y)|^{p}-|f(y)|^{p}-|f_i(y)-f(y)|^{p}\big|{\rm d}y\rightarrow 0,\quad \text{as}\ \ i\rightarrow\infty. \]

It follows that

(3.3)\begin{equation} \begin{aligned} \|f_i-f\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)} & =\|f_i\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)}-\|f\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)}+o(1)\\ & =1-\|f\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)}+o(1). \end{aligned} \end{equation}

On the other hand, since $Tf_i(x)\rightarrow Tf(x)$ for $x\in \mathbb {R}^{n}_+$ and $\|Tf_i\|_{L^{q}(\mathbb {R}^{n}_+)}\leq C_{n,\alpha,\beta,p}$, we see

\begin{align*} \|Tf_i\|^{q}_{L^{q}(\mathbb{R}^{n}_+)}& =\|Tf\|^{q}_{L^{q}(\mathbb{R}^{n}_+)}-\|Tf_i-Tf\|^{q}_{L^{q}(\mathbb{R}^{n}_+)}+o(1)\\ & \leq C^{q}_{n,\alpha,\beta, p}\|f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+ C^{q}_{n,\alpha,\beta, p}\|f_i-f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+o(1). \end{align*}

Hence,

(3.4)\begin{equation} 1\leq \|f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+ \|f_i-f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+o(1). \end{equation}

By (3.3) and (3.4) and letting $i\rightarrow \infty$, we derive

\[ 1\leq \|f\|^{q}_{L^{p}(\partial\mathbb{R}^{n}_+)}+ (1-\|f\|^{p}_{L^{p}(\partial\mathbb{R}^{n}_+)})^{{q}/{p}}. \]

Since $q>p$ and $f\neq 0$, we deduce that $\|f\|_{ L^{p}(\partial \mathbb {R}^{n}_+)}=1$. Hence, $f_i\rightarrow f$ in $L^{p}(\partial \mathbb {R}^{n}_+)$ and $f$ is a maximizer. This implies the existence of an extremal function.

Assume $f\in L^{p}(\partial \mathbb {R}^{n}_+)$ is a maximizer, then so is $|f|$. Hence $\|Tf\|_{L^{q}(\mathbb {R}^{n}_+)}=\|T|f|\|_{L^{q}(\mathbb {R}^{n}_+)}$, which implies either $f\geq 0$ or $f\leq 0$. Without loss of generality, we only consider the case of $f\geq 0$, then the Euler–Lagrange equation after scaling by a positive constant is given by equation (3.1)

(3.5)\begin{equation} f^{p-1}(y)=\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{(|x'-y|^{2}+x_n^{2})^{{(n-\alpha)}/{2}}}(Tf(x))^{q-1}{\rm d}x. \end{equation}

On the other hand, for $x_n>0$,

\[ \|K_{x_n}*f\|_{L^{q}(\mathbb{R}^{n}_+)}=\|K_{x_n}*f^{*}\|_{L^{q}(\mathbb{R}^{n}_+)}. \]

By (3.2), we deduce that

\[ f(y)=f^{*}(y-y_0)=f^{*}(|y-y_0|), \]

for some $y_0\in \partial \mathbb {R}^{n}_+$. It follows from the Euler–Lagrange equation (3.5) and lemma 2.2 of Lieb [39] that $f$ must be strictly decreasing along the radial direction.

4. The proof of theorem 4.1

In this section, we establish the regularity properties of solutions to the following Euler–Lagrange equation:

(4.1)\begin{equation} f^{p-1}(y)=\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta}}{(|x'-y|^{2}+x_n^{2})^{{(n-\alpha)}/{2}}}(Tf(x))^{q-1}{\rm d}x. \end{equation}

We prove

Theorem 4.1 Let $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta > 1$ and $1 < p< \frac {n-1}{\alpha +\beta -1}$. Suppose that $f\in L_{loc}^{p}(\partial \mathbb {R}^{n}_+)$ is nonnegative solution to (4.1) with $\frac {1}{q}=\frac {n-1}{n}(\frac {1}{p}-\frac {\alpha +\beta -1}{n-1})$. Then $f\in C^{\infty }(\partial \mathbb {R}^{n}_+)$.

Let $u(y)=f^{p-1}(y)$, $v(x)=Tf(x)$, $\theta =\frac {1}{p-1}$ and $\kappa =q-1$. Then Euler–Lagrange equation (4.1) can be rewritten as the following integral system

(4.2)\begin{equation} \begin{cases} u(y)=\displaystyle\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}} v^{\kappa}(x){\rm d}x, & y\in\partial\mathbb{R}^{n}_+,\\ v(x)=\displaystyle\int_{\partial\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}} u^{\theta}(y){\rm d}y, & x\in\mathbb{R}^{n}_+, \end{cases} \end{equation}

with $\frac {1}{\kappa +1}=\frac {n-1}{n}(\frac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$. If $f\in L_{loc}^{p}(\partial \mathbb {R}^{n}_+)$, then $u\in L_{loc}^{\theta +1}(\partial \mathbb {R}^{n}_+)$. Therefore, to prove theorem 4.1, it is sufficient to prove the following lemma.

Lemma 4.2 Assume that $\beta \geq 0,$ $\alpha +\beta > 1$ and $\frac {\alpha +\beta -1}{n-1}< \theta < \infty,$ and $0<\kappa <\infty$ given by

\[ \frac{1}{\kappa+1}=\frac{n-1}{n}\left(\frac{n-\alpha-\beta}{n-1}-\frac{1}{\theta+1}\right). \]

Suppose that $(u,\,v)$ is a pair of nonnegative solutions of (4.2) with $u\in L_{loc}^{\theta +1}(\partial \mathbb {R}^{n}_+)$. Then $u\in C^{\infty }(\partial \mathbb {R}^{n}_+)$ and $v\in C^{\infty }(\overline {\mathbb {R}^{n}_+})$.

To prove lemma 4.2, we first establish two local regularity results, which are spirited by Brezis and Kato's lemma A.1 in [5], Hang et al.'s propositions 5.2 and 5.3 in [33], Li's theorem 1.3 in [44], Dou and Zhu's propositions 4.3 and 4.4 in [22].

For $R>0$, define

\begin{align*} & B_R(x)=\{y\in \mathbb{R}^{n} \mid |y-x|< R,\ x\in \mathbb{R}^{n}\},\\ & B_R^{n-1}(x)=\{y\in \partial\mathbb{R}^{n}_+{\mid} |y-x|< R,\ x\in \partial\mathbb{R}^{n}_+\},\\ & B_R^{+}(x)=\{y=(y_1,y_2,\ldots,y_n)\in B_R(x) \mid y_n>0,\ x\in \partial\mathbb{R}^{n}_+\}. \end{align*}

For $x = 0$, we write

\[ B_R=B_R(0),\quad B_R^{n-1}=B_R^{n-1}(0),\ B_R^{+}=B_R^{+}(0). \]

Lemma 4.3 Assume that $\alpha +\beta >1,$ $1< a,\,b\leq \infty,$ $1\leq r<\infty,$ and $\frac {n}{n-\alpha -\beta }< p< q<\infty$ satisfy

(4.3)\begin{equation} \frac{\alpha+\beta}{n}<\frac{r}{q}+\frac{1}{a}<\frac{r}{p}+\frac{1}{a}<1, \frac{n}{ar}+\frac{n-1}{b}=\frac{\alpha+\beta}{r}+(\alpha+\beta-1). \end{equation}

Suppose that $v,\,h\in L^{p}(B^{+}_R)$, $V\in L^{a}(B_R^{+})$, and $U\in L^{b}(B_R^{n-1})$ are all nonnegative functions with $h|_{B^{+}_{R/2}}\in L^{q}(B^{+}_{R/2})$, and

\[ v(x)\leq\int_{B_R^{n-1}}\frac{x_n^{\beta} U(y)}{|x-y|^{n-\alpha}}\Big[ \int_{B_R^{+}}\frac{z_n^{\beta} V(z)v^{r}(z)}{|z-y|^{n-\alpha}}{\rm d}z\Big]^{{1}/{r}}{\rm d}y+h(x),\ \forall x\in B_R^{+}. \]

There is a $\epsilon =\epsilon (n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b)>0$, and $C=C(n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b,\,\epsilon )>0$ such that if

\[ \|U\|_{L^{b}(B_R^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B_R^{+})}\leq \epsilon(n,\alpha,\beta,p,q,r,a,b), \]

then,

\[ \|v\|_{L^{q}(B^{+}_{R/4})}\leq C(n,\alpha,\beta,p,q,r,a,b,\epsilon)\left(R^{{n}/{q}-{n}/{p}} \|v\|_{L^{p}(B^{+}_{R})}+\|h\|_{L^{q}(B^{+}_{R/2})}\right). \]

Proof. By scaling, we may assume $R = 1$. Assume that $v,\, h\in L^{q}(B^{+}_1)$. For $y\in B_1^{n-1}$, denote

\[ u(y)=\int_{B_1^{+}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x. \]

Let $p_1$ and $q_1$ be the numbers defined by

(4.4)\begin{equation} \frac{1}{p_1}=\frac{n}{n-1}\left(\frac{r}{p}+\frac{1}{a}-\frac{\alpha+\beta}{n}\right),\quad \frac{1}{q_1}=\frac{n}{n-1}\left(\frac{r}{q}+\frac{1}{a}-\frac{\alpha+\beta}{n}\right). \end{equation}

Then, it follows from inequality (1.6) that

(4.5)\begin{align} & \|u\|_{L^{p_1}(B^{n-1}_1)}\leq C(n,\alpha,\beta,p,r,a,b,\epsilon)\|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{p}(B^{+}_{1})}, \end{align}

(4.6)\begin{align} & \|u\|_{L^{q_1}(B^{n-1}_1)}\leq C(n,\alpha,\beta,q,r,a,b,\epsilon)\|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{q}(B^{+}_{1})}. \end{align}

Given $0<\delta _1<\delta _2\leq \frac {1}{2}$, for $x\in B^{+}_{\delta _2}$, we have

\begin{align*} v(x)& \leq\int_{B_{{(\delta_1+\delta_2)}/{2}}^{n-1}}\frac{ x_n^{\beta} U(y)u^{{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y +\int_{B_1^{n-1}\setminus{B_{{(\delta_1+\delta_2)}/{2}}^{n-1}}}\frac{ x_n^{\beta} U(y)u^{{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y+h(x)\\ & :=I_1(x)+I_2(x)+h(x). \end{align*}

By (4.3) and (4.4), we deduce that

\[ \frac{1}{q}=\frac{n-1}{n}\left(\frac{1}{b}+\frac{1}{q_1r}-\frac{\alpha+\beta-1}{n-1}\right), \]

which combines with (1.5) and the Hölder inequality, it yields that

\[ \|I_1\|_{L^{q}(B_{\delta_1}^{+})}\leq C(n,\alpha,\beta,p,r,a,b) \|U\|_{L^{b}(B_1^{n-1})} \|u\|^{{1}/{r}}_{L^{q_1}(B^{n-1}_{{(\delta_1+\delta_2)}/{2}})}. \]

Since $p>\frac {n}{n-\alpha -\beta }$, it follows from the Hölder inequality and (4.5) that

\begin{align*} I_2(x)& \leq \frac{C(n,\alpha,\beta)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|U\|_{L^{b}(B_1^{n-1})}\|u\|^{{1}/{r}}_{L^{p_1}(B^{n-1}_1)}\\ & \leq \frac{C(n,\alpha,\beta,p,r,a,b)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|U\|_{L^{b}(B_1^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B^{+}_1)}\|v\|_{L^{p}(B_1^{+})}. \end{align*}

Then, we have

(4.7)\begin{align} \begin{aligned} \|v\|_{L^{q}(B_{\delta_1}^{+})} & \leq C(n,\alpha,\beta,p,r,a,b) \|U\|_{L^{b}(B_1^{n-1})}\|u\|^{{1}/{r}}_{L^{q_1}(B^{n-1}_{{(\delta_1+\delta_2)}/{2}})}\\ & \quad+ \frac{C(n,\alpha,\beta,p,r,a,b)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|U\|_{L^{b}(B_1^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B^{+}_1)}\|v\|_{L^{p}(B_1^{+})} +\|h\|_{L^{q}(B^{+}_{{1}/{2}})}. \end{aligned} \end{align}

On the other hand, for $y\in B_{{(\delta _1+\delta _2)}/{2}}^{n-1}$, we derive

\begin{align*} u(y)& = \int_{B_{\delta_2}^{+}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x+\int_{B_1^{+}\setminus{B_{\delta_2}^{+}}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x\\ & \leq\int_{B_{\delta_2}^{+}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x+\frac{C(n,\alpha,\beta)}{(\delta_2-\delta_1)^{n-\alpha-\beta}}\int_{B_1^{+}\setminus{B_{\delta_2}^{+}}}V(x)v^{r}(x){\rm d}x\\ & \leq\int_{B_{\delta_2}^{+}}\frac{x_n^{\beta} V(x)v^{r}(x)}{|x-y|^{n-\alpha}}{\rm d}x+\frac{C(n,\alpha,\beta,a,b,p,r)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{p}(B^{+}_{1})}. \end{align*}

Combining this and inequality (4.6), we obtain

(4.8)\begin{equation} \begin{aligned}\|u\|_{L^{q_1}(B^{n-1}_{{(\delta_1+\delta_2)}/{2}})} & \leq C(n,\alpha,\beta,a,b,p,r) \|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{q}(B^{+}_{1})}\\ & \quad+ \frac{C(n,\alpha,\beta,a,p,r)}{(\delta_2-\delta_1)^{n-\alpha-\beta}} \|V\|_{L^{a}(B^{+}_{1})}\|v\|^{r}_{L^{p}(B^{+}_{1})}. \end{aligned} \end{equation}

By (4.7) and (4.8), we see

\begin{align*} \|v\|_{L^{q}(B_{\delta_1}^{+})}& \leq C(n,\alpha,\beta,p,r,a,b,\epsilon)\left(\frac{1}{(\delta_2-\delta_1)^{n-\alpha-\beta}} +\frac{1}{(\delta_2-\delta_1)^{n-\alpha-\beta}}\right)\|v\|_{L^{p}(B_1^{+})}\\ & \quad+ \frac{1}{2}\|v\|_{L^{q}(B_{\delta_2}^{+})}+\|h\|_{L^{q}(B^{+}_{{1}/{2}})}, \end{align*}

if $\epsilon$ is small enough. One can employ the usual iteration procedure (see [32]) to obtain

(4.9)\begin{equation} \|v\|_{L^{q}(B_{{1}/{4}}^{+})}\leq C(n,\alpha,\beta,p,r,a,b,\epsilon) \big(\|v\|_{L^{p}(B_1^{+})}+\|h\|_{L^{q}(B^{+}_{{1}/{2}})}\big). \end{equation}

For $v,\, h\in L^{p}(B^{+}_1)$, we will show inequality (4.9) still holds. Let $0\leq \eta (x)\leq 1$ be the measurable function such that

\[ v(x)\leq\eta(x)\int_{B_1^{n-1}}\frac{x_n^{\beta} U(y)}{|x-y|^{n-\alpha}}\Big[ \int_{B_1^{+}}\frac{z_n^{\beta} V(z)v^{r}(z)}{|z-y|^{n-\alpha}}{\rm d}z\Big]^{{1}/{r}}{\rm d}y+\eta(x)h(x),\quad \forall x\in B_1^{+}. \]

Define a map $T_1$ by

\[ T_1(\varphi)(x)\leq\eta(x)\int_{B_1^{n-1}}\frac{x_n^{\beta} U(y)}{|x-y|^{n-\alpha}}\left[ \int_{B_1^{+}}\frac{z_n^{\beta} V(z)|\varphi(z)|^{r}}{|z-y|^{n-\alpha}}{\rm d}z\right]^{{1}/{r}}{\rm d}y. \]

Choosing small enough $\epsilon (n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b)$, in view of the integral inequality (1.5), we have

\begin{align*} & \|T_1(\varphi)\|_{L^{p}(B_{1}^{+})}\\& \leq C(n,\alpha,\beta,p,r,a,b) \|U\|_{L^{b}(B_1^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B^{+}_1)}\|\varphi\|_{L^{p}(B_1^{+})}\leq\frac{1}{2}\|\varphi\|_{L^{p}(B_1^{+})},\\ & \|T_1(\varphi)\|_{L^{q}(B_{1}^{+})}\\& \leq C(n,\alpha,\beta,p,r,a,b) \|U\|_{L^{b}(B_1^{n-1})}\|V\|^{{1}/{r}}_{L^{a}(B^{+}_1)}\|\varphi\|_{L^{q}(B_1^{+})}\leq\frac{1}{2}\|\varphi\|_{L^{q}(B_1^{+})}. \end{align*}

Furthermore, one can utilize the Minkowski inequality to obtain that for $\varphi,\, \psi \in L^{p}(B^{+}_1)$,

\[ |T_1(\varphi)(x)-T_1(\psi)(x)|\leq T_1(|\varphi-\psi|)(x),\ \ x\in B^{+}_1, \]

which implies

\[ \|T_1(\varphi)-T_1(\psi)\|_{L^{p}(B_{1}^{+})}\leq \|T_1(|\varphi-\psi|)\|_{L^{p}(B_{1}^{+})}\leq \frac{1}{2}\|\varphi-\psi\|_{L^{p}(B_{1}^{+})}. \]

Similarly, we also obtain

\[ \|T_1(\varphi)-T_1(\psi)\|_{L^{q}(B_{1}^{+})}\leq \frac{1}{2}\|\varphi-\psi\|_{L^{q}(B_{1}^{+})}. \]

for any $\varphi,\, \psi \in L^{q}(B^{+}_1)$.

Set $h_j(x)=\min \{v(x),\, j\}$, using the regular lifting theorem with contracting operators which can be seen in [16, 46], we may find a unique $u_j\in L^{q}(B^{+}_1)$ such that

\begin{align*} v_j(x)& =T_1(v_j)(x)+\eta(x)h_j(x)\\ & =\eta(x)\int_{B_1^{n-1}}\frac{x_n^{\beta} U(y)}{|x-y|^{n-\alpha}}\left[ \int_{B_1^{+}}\frac{z_n^{\beta} V(z)v_j^{r}(z)}{|z-y|^{n-\alpha}}{\rm d}z\right]^{{1}/{r}}{\rm d}y+\eta(x)h_j(x),\quad \forall x\in B_1^{+}. \end{align*}

Applying a priori estimate to $v_j$, we obtain

(4.10)\begin{equation} \|v_j\|_{L^{q}(B_{{1}/{4}}^{+})}\leq C(n,\alpha,\beta,p,r,a,b,\epsilon) \big(\|v_j\|_{L^{p}(B_1^{+})}+\|h_j\|_{L^{q}(B^{+}_{{1}/{2}})}\big). \end{equation}

Observing that

\[ v(x)=T_1(v)(x)+\eta(x)h(x), \]

then we see that

\begin{align*} \|v_j-v\|_{L^{p}(B_{1}^{+})}& \leq\|T_1(v_j)-T_1(v)\|_{L^{p}(B_{1}^{+})}+\|h_j-h\|_{L^{p}(B_{1}^{+})}\\ & \leq\frac{1}{2}\|v_j-v\|_{L^{p}(B_{1}^{+})}+\|h_j-h\|_{L^{p}(B_{1}^{+})}. \end{align*}

Hence,

\[ \|v_j-v\|_{L^{p}(B_{1}^{+})}\leq 2\|h_j-h\|_{L^{p}(B_{1}^{+})}\rightarrow 0, \text{ as } j\rightarrow\infty. \]

Taking a limit process in inequality (4.10), we conclude that

\[ \|v\|_{L^{q}(B_{{1}/{4}}^{+})}\leq C(n,\alpha,\beta,p,r,a,b,\epsilon)\left(\|v\|_{L^{p}(B_1^{+})}+\|h\|_{L^{q}(B^{+}_{{1}/{2}})}\right). \]

This completes the proof of lemma 4.3.

Similarly, we also can obtain the following local regularity lemma.

Lemma 4.4 Assume that $\alpha +\beta >1,$ $1< a,\,b\leq \infty,$ $1\leq r<\infty,$ and $\tfrac {n-1}{n-\alpha -\beta }< p< q<\infty$ satisfy

(4.11)\begin{equation} \frac{\alpha+\beta-1}{n-1}<\frac{r}{q}+\frac{1}{a}<\frac{r}{p}+\frac{1}{a}<1,\quad \frac{n-1}{ar}+\frac{n}{b}=\frac{\alpha+\beta-1}{r}+(\alpha+\beta). \end{equation}

Suppose that $u,\,g\in L^{p}(B^{n-1}_R),$ $V\in L^{b}(B_R^{+})$ and $U\in L^{a}(B_R^{n-1})$ are all nonnegative functions with $g|_{B^{n-1}_{R/2}}\in L^{q}(B^{n-1}_{R/2}),$ and

\[ u(y)\leq\int_{B_R^{+}}\frac{x_n^{\beta} V(x)}{|x-y|^{n-\alpha}}\left[ \int_{B_R^{n-1}}\frac{x_n^{\beta} U(z)u^{r}(z)}{|z-x|^{n-\alpha}}{\rm d}z\right]^{{1}/{r}}{\rm d}x+g(y),\quad\forall y\in B_R^{n-1}. \]

There is a $\epsilon =\epsilon (n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b)>0$, and $C=C(n,\,\alpha,\,\beta,\,p,\,q,\,r,\,a,\,b,\,\epsilon )>0$ such that if

\[ \|U\|^{{1}/{r}}_{L^{b}(B_R^{n-1})}\|V\|_{L^{a}(B_R^{+})}\leq \epsilon(n,\alpha,\beta,p,q,r,a,b), \]

then,

\begin{align*} & \|u\|_{L^{q}(B^{n-1}_{R/4})}\\& \quad \leq C(n,\alpha,\beta,p,q,r,a,b,\epsilon)\big(R^{{(n-1)}/{q}-{(n-1)}/{p}} \|u\|_{L^{p}(B^{n-1}_{R})}+\|g\|_{L^{q}(B^{n-1}_{R/2})}\big). \end{align*}

Based on lemmas 4.3 and 4.4, we prove lemma 4.2. For $R>0$, define

\[ u_R(y)=\int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y|^{n-\alpha}}{\rm d}x,\quad v_R(x)=\int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{x_n^{\beta} u^{\theta}(y)}{|x-y|^{n-\alpha}}{\rm d}y. \]

By (4.2), we have

\[ u(y)=\int_{B_R^{+}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y|^{n-\alpha}}{\rm d}x+u_R(y),\quad v(x)=\int_{B_R^{n-1}}\frac{x_n^{\beta} u^{\theta}(y)}{|x-y|^{n-\alpha}}{\rm d}y+v_R(x). \]

We first verify that if $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$, then

\[ v\in L^{\kappa+1}_{loc}(\overline{\mathbb{R}^{n}_+}),\quad v_R\in L^{\infty}_{loc} (B_R^{+}\cup B_R^{n-1}). \]

Indeed, since $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$, we see $u<\infty$, a.e. on $\partial \mathbb {R}^{n}_+$. This implies $v<\infty$, a.e. on $\mathbb {R}^{n}_+$. Hence there exists an $x^{0}=(x_1^{0},\,x_2^{0},\,\ldots,\,x_n^{0})\in B_R^{+}$ and $x_n^{0}>\frac {R}{4}$ such that $v(x^{0})<\infty$. It follows that

\begin{align*} \int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{ u^{\theta}(y)}{|y|^{n-\alpha}}{\rm d}y & \leq c\int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{(x^{0}_n)^{\beta} u^{\theta}(y)}{|x^{0}-y|^{n-\alpha}}{\rm d}y\\ & \leq c v(x^{0})<\infty. \end{align*}

For $0 <\delta < 1$, $x \in B^{+}_{\delta R}$, it holds,

\[ v_R(x)\leq\frac{c R^{\beta}}{(1-\delta)^{n-\alpha}} \int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{ u^{\theta}(y)}{|y|^{n-\alpha}}{\rm d}y, \]

which implies that

\[ v_R\in L^{\infty}_{loc} (B_R^{+}\cup B_R^{n-1}). \]

Thanks to the integral inequality (1.5) with $\frac {1}{\kappa +1}=\frac {n-1}{n}(\frac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$, we derive that

\[ \left[\int_{\mathbb{R}^{n}_+}\left( \int_{B_R^{n-1}}\frac{x_n^{\beta} u^{\theta}(y)}{|x-y|^{n-\alpha}}{\rm d}y \right)^{\kappa+1}{\rm d}x\right]^{{1}/{(\kappa+1)}} \leq \|u\|^{\theta}_{L^{\theta+1}(B^{n-1}_R)}<\infty. \]

Hence,

\[ v\in L^{\kappa+1}_{loc}(B_R^{+}\cup B_R^{n-1}). \]

Since $R$ is arbitrary, we deduce that

\[ v\in L^{\kappa+1}_{loc}(\overline{\mathbb{R}^{n}_+}). \]

We now turn to verify that $u_R\in L^{\infty }_{loc}(B^{n-1}_R).$ Since $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$, there is a $y_0\in B^{n-1}_{{R}/{2}}$ such that $u(y_0)<\infty$. Thus,

\begin{align*} \int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x|^{n-\alpha}}{\rm d}x & \leq c\int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y_0|^{n-\alpha}}{\rm d}x\\ & \leq c u(y_0)<\infty. \end{align*}

For $0 <\delta < 1$, $x \in B^{n-1}_{\delta R}$, one can calculate that

\[ u_R(y)=\frac{c }{(1-\delta)^{n-\alpha}} \int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x|^{n-\alpha}}{\rm d}x<\infty, \]

which leads to $u_R\in L^{\infty }_{loc}(B^{n-1}_R).$

To prove the regularity of $u$, we discuss two cases.

Case 1. $\frac {\alpha +\beta -1}{n-\alpha -\beta }<\theta <\frac {n+\alpha +\beta -2}{n-\alpha -\beta }$.

Since $\frac {1}{\kappa +1}=\frac {n-1}{n}(\tfrac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$ and $\theta <\frac {n+\alpha +\beta -2}{n-\alpha -\beta }$, we have $\kappa >\frac {n+\alpha +\beta }{n-\alpha -\beta }$. Then one can deduce that

\[ \kappa-\frac{\alpha+\beta}{n}(\kappa+1)>\frac{1}{\theta}, \quad \text{and } \kappa-\frac{\alpha+\beta}{n}(\kappa+1)>1. \]

Hence, we choose a fixed number $r$ such that

\[ 1<\kappa-\frac{\alpha+\beta}{n}(\kappa+1)\leq r\leq\kappa, \text{ and } r>\frac{1}{\theta}, \]

then it follows that

\[ u^{{1}/{r}}(y)\leq \left(\int_{B_R^{+}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y|^{n-\alpha}}{\rm d}x\right)^{{1}/{r}} +u^{{1}/{r}}_R(y). \]

Then,

\[ v(x)\leq\int_{B_R^{n-1}}\frac{ x_n^{\beta} u^{\theta-{1}/{r}}(y)}{|x-y|^{n-\alpha}} \left(\int_{B_R^{+}}\frac{z_n^{\beta} v^{\kappa-r}(z)v^{r}(z)}{|z-y|^{n-\alpha}}{\rm d}z\right)^{{1}/{r}}{\rm d}y +h_R(x), \]

where

\[ h_R(x)=\int_{B_R^{n-1}}\frac{ x_n^{\beta} u^{\theta-{1}/{r}}(y)u_R^{{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y +v_R(x). \]

Since $u\in L^{\infty }_{loc}(\partial \mathbb {R}^{n}_+)$, for any $x\in B^{+}_R$, it holds,

\[ \int_{B_R^{n-1}}\frac{ x_n^{\beta} u^{\theta-{1}/{r}}(y)u_R^{{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y \leq \|u_R\|_{L^{\infty}(B^{n-1}_R)} \int_{B^{n-1}_R} \frac{ x_n^{\beta} u^{\theta-{1}/{r}}(y)}{|x-y|^{n-\alpha}}{\rm d}y. \]

It follows from inequality (1.5) and $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$ that

\[ h_R\in L^{q_0}(B_R^{+}\cup B_R^{n-1}), \]

where $\frac {1}{q_0}=\frac {1}{\kappa +1}-\frac {n-1}{n}\frac {1}{r(\theta +1)}.$ For $\epsilon > 0$ small enough, one can choose $\kappa -\frac {\alpha +\beta }{n}(\kappa +1)+\epsilon >1+\epsilon$ such that

\[ q_0=\frac{rn(\kappa+1)}{rn-(k+1)(n-1)({1}/{(\theta+1)})}=\frac{rn(\kappa+1)}{n\epsilon}>\frac{\kappa+1}{\epsilon} \]

can be any large number when we choose $\epsilon$ small enough. Hence, it follows that $h_R\in L^{q}(B_R^{+}\cup B_R^{n-1})$ for any $q<\infty$.

Let

\[ a=\frac{\kappa+1}{\kappa-r},\quad b=\frac{\theta+1}{\theta-{1}/{r}}, \ p=\kappa+1>\frac{n}{n-\alpha-\beta}, \]

which combines with $\frac {1}{\kappa +1}=\frac {n-1}{n}(\frac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$, we obtain

\[ \frac{n}{ar}+\frac{n-1}{b}=\frac{\alpha+\beta}{r}+(\alpha+\beta-1),\quad \frac{r}{p}+\frac{1}{a}=\frac{\kappa}{\kappa+1}<1. \]

Since $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$ and $v\in L^{\kappa +1}_{loc}(\overline {\mathbb {R}^{n}_+})$, one can choose $q$ such that $q\in (\kappa +1,\,\tfrac {rn(\kappa +1)}{(\alpha +\beta )(\kappa +1)-n(k-r)})$, then it is easy to check that $\tfrac {r}{q}+\tfrac {1}{a}>\tfrac {\alpha +\beta }{n}$. It follows from lemma 4.3 that $v|_{B_{{R}/{4}}^{+}}\in L^{q}({B^{+}_{{R}/{4}}})$. Notice that $\frac {n\kappa }{\alpha +\beta }<\tfrac {rn(\kappa +1)}{(\alpha +\beta )(\kappa +1)-n(k-r)}$. For $q\in (\tfrac {n\kappa }{\alpha +\beta },\,\tfrac {rn(\kappa +1)}{(\alpha +\beta )(\kappa +1)-n(k-r)})$, we have

\begin{align*} u(y)& \leq R^{\beta}\left(\int_{B^{+}_{{R}/{4}}}|x-y|^{{((\alpha-n)q)}/{(q-\kappa)}}{\rm d}x\right)^{{(q-\kappa)}/{q}} \|v\|^{\kappa}_{L^{q}(B^{+}_{{R}/{4}})}+u_{{R}/{4}}(y)\\ & \leq cR^{\alpha+\beta-n+({(n(q-k))}/{q})}\|v\|^{\kappa}_{L^{q}(B^{+}_{{R}/{4}})}+u_{{R}/{4}}(y)<\infty, \end{align*}

which implies that

\[ u|_{B^{n-1}_{{R}/{8}}}\in L^{\infty}(B^{n-1}_{{R}/{8}}). \]

Since every point may be viewed as a centre, we see $u\in L^{\infty }_{loc}(\partial \mathbb {R}^{n}_+)$, and hence $v\in L^{\infty }_{loc}(\overline {\mathbb {R}^{n}_+})$.

For any $R>0$, one can apply

\[ \int_{\partial\mathbb{R}^{n}_+{\setminus}{B_R^{n-1}}}\frac{ u^{\theta}(y)}{|y|^{n-\alpha}}{\rm d}y<\infty, \text{ and } \int_{\mathbb{R}^{n}_+{\setminus}{B_R^{+}}}\frac{x_n^{\beta} v^{\kappa}(x)}{|x-y_0|^{n-\alpha}}{\rm d}x<\infty \]

to obtain $v_R \in C^{\infty }(B^{+}_R\cup B_R^{n-1})$ and $u_R \in C^{\infty }(B_R^{n-1})$ which yields that $u \in C^{\gamma }_{loc}(\partial \mathbb {R}^{n}_+)$ for $0 <\gamma < 1$. By the standard potential theory (see [29], chap. 4) and bootstrap method, we see that $(u,\, v) \in C^{\infty }(\partial \mathbb {R}^{n}_+)\times C^{\infty }(\overline {\mathbb {R}^{n}_+})$.

Case 2. $\frac {n+\alpha +\beta -2}{n-\alpha -\beta }\leq \theta <\infty$.

Since $\frac {1}{\kappa +1}=\frac {n-1}{n}(\tfrac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$, it is easy to check that

\[ \theta-\frac{\alpha+\beta-1}{n-1}(\theta+1)>\frac{1}{\kappa},\text{ and } \theta-\frac{\alpha+\beta-1}{n-1}(\theta+1)\geq 1. \]

Choosing a fixed number $r$ satisfying

\[ 1\leq\theta-\frac{\alpha+\beta-1}{n-1}(\theta+1)\leq r\leq\theta, \text{ and } r>\frac{1}{\kappa}, \]

then it follows that

\[ v^{{1}/{r}}(x)\leq \left(\int_{B_R^{n-1}}\frac{x_n^{\beta} u^{\theta}(y)}{|x-y|^{n-\alpha}}{\rm d}y\right)^{{1}/{r}} +v^{{1}/{r}}_R(x). \]

Hence,

\[ u(y)\leq\int_{B_R^{+}}\frac{ x_n^{\beta} v^{\kappa-{1}/{r}}(x)}{|x-y|^{n-\alpha}} \left(\int_{B_R^{n-1}}\frac{x_n^{\beta} u^{\theta}(z)}{|x-z|^{n-\alpha}}{\rm d}z\right)^{{1}/{r}}{\rm d}x +g_R(y), \]

where

\[ g_R(y)=\int_{B_R^{+}}\frac{ x_n^{\beta} v^{\kappa-{1}/{r}}(x)v_R^{{1}/{r}}(x)}{|x-y|^{n-\alpha}}{\rm d}x +u_R(y). \]

For any $y\in B^{n-1}_R$, it holds,

\[ \int_{B_R^{+}}\frac{ x_n^{\beta} v^{\kappa-{1}/{r}}(x)v_R^{{1}/{r}}(x)}{|x-y|^{n-\alpha}}{\rm d}x \leq \|v_R\|_{L^{\infty}(B^{+}_R)} \int_{B^{+}_R} \frac{ x_n^{\beta} v^{\kappa-{1}/{r}}(x)}{|x-y|^{n-\alpha}}{\rm d}x. \]

It follows from inequality (1.6) that $g_R\in L^{q_1}(B_R^{n-1})$ with $q_1$ given by

\[ \frac{1}{q_1}=\frac{1}{\theta+1}-\frac{n}{n-1}\frac{1}{r(\kappa+1)}. \]

Let

\[ a=\frac{\theta+1}{\theta-r},\quad b=\frac{\kappa+1}{\kappa-{1}/{r}}, \quad p=\theta+1>\frac{n-1}{n-\alpha-\beta}, \]

which combines with $\frac {1}{\kappa +1}=\frac {n-1}{n}(\tfrac {n-\alpha -\beta }{n-1}-\frac {1}{\theta +1})$, we obtain

\[ \frac{n-1}{ar}+\frac{n}{b}=\frac{\alpha+\beta-1}{r}+(\alpha+\beta),\quad \frac{r}{p}+\frac{1}{a}=\frac{\theta}{\theta+1}<1. \]

Since $u\in L^{\theta +1}_{loc}(\partial \mathbb {R}^{n}_+)$ and $v\in L^{\kappa +1}_{loc}(\overline {\mathbb {R}^{n}_+})$, one can choose $q$ such that

\[ q\in \left(\theta+1,\frac{r(n-1)(\theta+1)}{(\alpha+\beta-1)(\theta+1)-(n-1)(\theta-r)}\right), \]

then it is easy to check that $\frac {r}{q}+\frac {1}{a}>\frac {\alpha +\beta -1}{n-1}$. It follows from lemma 4.4 that $u|_{B^{n-1}_{{R}/{4}}}\in L^{q}({B^{n-1}_{{R}/{4}}})$. Arguing this as we did in case 1, and by the standard bootstrap method, we conclude that $(u,\, v) \in C^{\infty }(\partial \mathbb {R}^{n}_+)\times C^{\infty }(\overline {\mathbb {R}^{n}_+})$.

5. The proof of theorem 1.7

In this section, we investigate the necessary and sufficient condition for the existence of nonnegative nontrivial solutions to the following integral system:

(5.1)\begin{equation} \begin{cases} u(y)=\displaystyle\int_{\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}} v^{\kappa}(x){\rm d}x, & y\in\partial\mathbb{R}^{n}_+,\\ v(x)=\displaystyle\int_{\partial\mathbb{R}^{n}_+}\frac{x_n^{\beta} }{|x-y|^{n-\alpha}}u^{\theta}(y) {\rm d}y, & x\in\mathbb{R}^{n}_+. \end{cases} \end{equation}

From theorems 3.1 and 4.1, to obtain the proof of theorem 1.7, it is sufficient to prove the following theorem.

Theorem 5.1 For $n\geq 2,$ $\beta \geq 0,$ $\alpha +\beta > 1,$ $\theta >0,$ $\kappa >0,$ assume that $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ is a pair of nonnegative nontrivial $C^{1}$ solutions of (5.1), then a necessary condition for $\theta$ and $\kappa$ is

\[ \frac{n-1}{\theta+1}+\frac{n}{\kappa+1}=n-\alpha-\beta. \]

Proof. Assume that $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ is a pair of nonnegative nontrivial solutions of the integral system (5.1). One can apply the integration by parts to obtain

\begin{align*} & \int_{B_R^{n-1}} u^{\theta}(y)(y\nabla u(y)){\rm d}y\\ & \quad= \frac{1}{\theta+1}\int_{B_R^{n-1}} y\nabla (u^{\theta+1}(y)){\rm d}y\\ & \quad= \frac{R}{\theta+1}\int_{\partial B_R^{n-1}}u^{\theta+1}(y){\rm d}\sigma-\frac{n-1}{\theta+1}\int_{B_R^{n-1}} u^{\theta+1}(x){\rm d}x. \end{align*}

Similarly, one can also derive that

\begin{align*} & \int_{B_R^{+}} v^{\kappa}(x)(x\nabla v(x)){\rm d}x\\ & \quad = \frac{R}{\kappa+1}\int_{\{\partial B_R^{+}\cap x_n>0\}}v^{\kappa+1}(x){\rm d}\sigma-\frac{n}{\kappa+1}\int_{B_R^{+}} v^{\kappa+1}(x){\rm d}x.\\ \end{align*}

It follows from $(u,\,v)\in L^{\theta +1}(\partial \mathbb {R}^{n}_+)\times L^{\kappa +1}(\mathbb {R}^{n}_+)$ that there exists $R=R_j\rightarrow +\infty$ such that

\[ R_j\int_{\partial B_{R_j}^{n-1}} u^{\theta+1}(y){\rm d}\sigma\rightarrow 0,\quad R_j\int_{\{\partial B_{R_j}^{+}\cap x_n>0\}} v^{\kappa+1}(x){\rm d}\sigma\rightarrow 0. \]

Therefore, we get

(5.2)\begin{equation} \begin{aligned} & \int_{\partial\mathbb{R}^{n}_+} u^{\theta}(y)(y\nabla u(y)){\rm d}y+\int_{\mathbb{R}^{n}_+} v^{\kappa}(x)(x\nabla v(x)){\rm d}x\\ & \quad={-}\frac{n-1}{1+\theta}\int_{\partial\mathbb{R}^{n}_+} u^{1+\theta}(x){\rm d}x-\frac{n}{1+\kappa}\int_{\mathbb{R}^{n}_+} v^{1+\kappa}(x){\rm d}x. \end{aligned} \end{equation}

On the other hand, one can calculate that

\begin{align*} \nabla u(y) y& =\frac{{\rm d}[u(\rho y)]}{{\rm d}\rho}\Big|_{\rho=1}\\ & ={-}(n-\alpha)\int_{\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n+2-\alpha}} [(y-x) y]v^{\kappa}(x){\rm d}x, \end{align*}

and

\begin{align*} \nabla v(x) x& =\frac{{\rm d}[v(\rho x)]}{{\rm d}\rho}\Bigg|_{\rho=1}\\ & ={-}(n-\alpha)\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n+2-\alpha}}[(y-x) x]u^{\theta}(y){\rm d}y\\ & \quad+ \beta\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n-\alpha}} u^{\theta}(y){\rm d}y. \end{align*}

It follows from Fubini's theorem that

\begin{align*} & \int_{\partial\mathbb{R}^{n}_+} u^{\theta}(y)(y\nabla u(y)){\rm d}y+\int_{\mathbb{R}^{n}_+} v^{\kappa}(x)(x\nabla v(x)){\rm d}x\\ & \quad= (\alpha+\beta-n)\int_{\mathbb{R}^{n}_+}\int_{\partial\mathbb{R}^{n}_+} \frac{x_n^{\beta} }{|x-y|^{n-\alpha}} u^{\theta}(y)v^{\kappa}(x){\rm d}y{\rm d}x\\ & \quad= (\alpha+\beta-n)\int_{\partial\mathbb{R}^{n}_+}u^{\theta+1}(y){\rm d}y\\ & \quad= (\alpha+\beta-n)\int_{\mathbb{R}^{n}_+}v^{\kappa+1}(x){\rm d}x. \end{align*}

This together with (5.2) implies that $\frac {n-1}{\theta +1}+\frac {n}{\kappa +1}=n-\alpha -\beta$.