Boundary maps and maximal representations on infinite-dimensional Hermitian symmetric spaces

Abstract

We define a Toledo number for actions of surface groups and complex hyperbolic lattices on infinite-dimensional Hermitian symmetric spaces, which allows us to define maximal representations. When the target is not of tube type, we show that there cannot be Zariski-dense maximal representations, and whenever the existence of a boundary map can be guaranteed, the representation preserves a finite-dimensional totally geodesic subspace on which the action is maximal. In the opposite direction, we construct examples of geometrically dense maximal representation in the infinite-dimensional Hermitian symmetric space of tube type and finite rank. Our approach is based on the study of boundary maps, which we are able to construct in low ranks or under some suitable Zariski density assumption, circumventing the lack of local compactness in the infinite-dimensional setting.

1 Introduction

1.1 Representations of semisimple Lie groups and their lattices

Lattices (that is, discrete subgroups with finite covolume) of semisimple Lie groups may be thought of as discretizations of these Lie groups. The question of knowing how much of the ambient group is encoded in its lattices is very natural and has attracted a lot of interest in the past decades.

Among the many results, one can highlight Mostow’s strong rigidity that implies that a lattice in a higher-rank semisimple Lie group without compact factors completely determines the Lie group [Mos73]. Later, Margulis proved his super-rigidity theorem and showed that linear representations of irreducible lattices of higher-rank semisimple algebraic groups over local fields are ruled by representations of the ambient algebraic groups [Mar91].

These rigidity results may be understood using a geometric object associated with the algebraic group: a Riemannian symmetric space (for a Lie group) or a Euclidean building (for an algebraic group over a non-Archimedean field).

Lattices have natural and interesting linear representations outside the finite-dimensional world, which starts with Hilbert spaces. For example, some representations may come from the principal series of the Lie group. Outside the world of unitary representations, some infinite-dimensional representations of a lattice have a very strong geometric flavor. This is the case when there is an invariant non-degenerate quadratic or Hermitian form of the finite index, that is, when the representation falls in $\operatorname {\mathrm{PO}}_{{\mathbf K}}(p,\infty )$ where ${\textbf K}=\textbf {R}$ or ${\mathbf{C}}$ and p is finite. Then, one can consider the associated action on some infinite-dimensional Riemannian symmetric space of non-positive curvature ${\mathcal X}_{{\mathbf K}}(p,\infty )$ . For example, when $p=1$ , ${\mathcal X}_{{\mathbf K}}(p,\infty )$ is the infinite-dimensional real or complex hyperbolic space. Gromov had the following expressive words to say about ${\mathcal X}_{\mathbf {R}}(p,\infty )$ [Gro93, p. 121]:

These spaces look as cute and sexy to me as their finite-dimensional siblings but they have been neglected by geometers and algebraists alike.

In [Duc15b], an analog of Margulis super-rigidity has been obtained for higher-rank cocompact lattices of semisimple Lie groups using harmonic map techniques. The main result is that non-elementary representations preserve a totally geodesic copy of a finite-dimensional symmetric space of non-compact type. The finite-rank assumption, here $p<\infty $ , may be thought of as a geometric Ersatz of local compactness.

The reader should be warned that even in the case of actions on finite-rank symmetric spaces of infinite dimension, some new baffling phenomena may appear. For example, Delzant and Py exhibited representations of $\operatorname {\textrm {PSL}}_2(\textbf {R})$ in $\operatorname {\textrm {O}}_{\mathbf {R}}(1,\infty )$ (and, more generally, of $\operatorname {\mathrm{PO}}(1,n)$ in $\operatorname {\textrm {O}}_{\mathbf {R}}(p,\infty )$ for some values of p depending on n). They found a one-parameter family of exotic deformations of ${\mathcal X}_{\mathbf {R}}(1,2)$ in ${\mathcal X}_{\mathbf {R}}(1,\infty )$ equivariant with respect to representations leaving no finite-dimensional totally geodesic subspace invariant. See [DP12, MP14] for a classification. Very recently, this classification has been extended to self-representations of $\operatorname {\textrm {O}}_{\mathbf {R}}(1,\infty )$ [MP18]. Moreover, exotic representations of $\operatorname {\mathrm{SU}}(1,n)$ in $\operatorname {\textrm {O}}_{\mathbf{C}}(1,\infty )$ have also been obtained in [Mon18].

In rank one, there is, in general, no hope for an analog of Margulis super-rigidity (even in finite dimension). For example, fundamental groups of non-compact hyperbolic surfaces of finite volume are free groups and thus not rigid. For compact hyperbolic surfaces, the lack of rigidity gives rise to the Teichmüller space and thus to a whole variety of deformations of the corresponding lattices.

For complex hyperbolic lattices, the complex structure constrains the lattices because the Kähler form implies the non-vanishing of the cohomology in degree two. Furthermore, in finite dimension, the Kähler form was successfully used to define a characteristic invariant that selects representations with surprising rigidity properties, the so-called Toledo invariant [BIW10, Tol89].

The goal of this paper is to study representations of complex hyperbolic lattices in the groups $\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ and $\operatorname {\mathrm{PO}}_{\mathbf {R}}(2,\infty )$ , and the associated isometric actions on the Hermitian symmetric spaces ${\mathcal X}_{\mathbf{C}}(p,\infty )$ and ${\mathcal X}_{\mathbf {R}}(2,\infty )$ . These spaces have a Kähler form $\omega $ and this yields a class in bounded cohomology of degree two on $G=\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ induced by the cocycle that computes the integral of the Kähler form $\omega $ over a straight geodesic triangle $\Delta (g_0x,g_1x,g_2x)$ whose vertices are in the orbit of a basepoint:

$$ \begin{align*}C_{\omega}^x(g_0,g_1,g_2)=\frac{1}{\pi}\int_{\Delta(g_0x,g_1x,g_2x)}\omega.\end{align*} $$

We denote by $\kappa ^b_G\in \text {H}_b^2(G,\textbf {R})$ the associated cohomology class where $G=\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ (see §5). As in finite dimension, the Gromov norm $\|\kappa ^b_G\|_{\infty }$ is exactly the rank of ${\mathcal X}_{\mathbf{C}}(p,\infty )$ (after normalization of the metric). Let $\rho \colon \Gamma \to \operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ be a homomorphism of a complex hyperbolic lattice. Pulling back $\kappa ^b_G$ by $\rho $ , one gets a bounded cohomology class for $\Gamma $ and one can define maximal representations of $\Gamma $ as representations maximizing a Toledo number defined as in finite dimension (see Definition 5.7).

Our main results concern maximal representations of fundamental groups of surfaces and, more generally, hyperbolic lattices. It is a continuation of previous results for finite-dimensional Hermitian targets, see [BI08, BIW10, KM17, Poz15] among other references. The meaning of Zariski density in infinite dimension is explained in the following subsection. For representations with target $\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ , we prove rigidity in the following way.

Theorem 1.1. Let ${\Gamma }<\operatorname {\mathrm{SU}}(1,n)$ be a complex hyperbolic lattice with n a positive integer, and let $\rho \colon \Gamma \to \operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ be a maximal representation. If $p\leq 2$ , then there is a finite-dimensional totally geodesic Hermitian symmetric subspace $\mathcal Y\subset {\mathcal X}_{\mathbf{C}}(p,\infty )$ that is invariant by $\Gamma $ . Furthermore, the representation $\Gamma \to \operatorname {\mathrm{Isom}}(\mathcal Y)$ is maximal.

More generally, for any $p\in \mathbf{N}$ , there is no maximal Zariski-dense representation $\rho \colon {\Gamma }\to \operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ .

In particular, because $\mathcal Y$ is finite dimensional, the results of Burger and Iozzi [BI08], the third author [Poz15] and Koziarz and Maubon [KM17] apply.

Interestingly enough, the analogous result of Theorem 1.1 does not hold for the orthogonal group $\operatorname {{\textrm O_{\mathbf {R}}}}(2,\infty )$ and $n=1$ . Let $\Sigma $ be a compact connected Riemann surface of genus one with one connected boundary component (which is a circle), that is, a one-holed torus. The fundamental group $\Gamma _{\Sigma }$ of $\Sigma $ is thus a free group on two generators and a lattice in $\operatorname {\mathrm{SU}}(1,1)$ .

Theorem 1.2. There are geometrically dense maximal representations $\rho :{\Gamma }_{\Sigma }\to \operatorname {\mathrm{PO}}_{\mathbf {R}}(2,\infty )$ .

Observe that the properties of maximal representations in $\operatorname {\mathrm{PO}}_{\mathbf {R}}(2,\infty )$ and $\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ are so different because, for every p, the Hermitian Lie group $\operatorname {{\textrm O_{\mathbf {R}}}}(2,p)$ is of tube type, while the Hermitian Lie groups $\operatorname {\mathrm{SU}}(p,q)$ are of tube type if and only if $p=q$ . We refer to §2.4 for more details. This allows much more flexibility, the chain geometry at infinity being trivial.

Not much is known about the representations of complex hyperbolic lattices, and even less so in infinite dimension. In the case of surface groups, instead, from the complementary series of $\operatorname {\textrm {PSL}}_2(\textbf {R})$ , Delzant and Py exhibited one-parameter families of representations in $\operatorname {\mathrm{PO}}_{\mathbf {R}}(p,\infty )$ for every $p\in \mathbf{N}$ [DP12]. Having explicit representations in $\operatorname {\mathrm{PO}}_{\mathbf {R}}(2,\infty )$ , it is compelling to determine if they induce maximal representations. Showing that some harmonic equivariant map is actually totally real, we conclude in Appendix A that the Toledo invariant of these representations vanishes.

Remark 1.3. The difference between $p\leq 2$ and $p>2$ lies in the hypotheses under which we can prove the existence of boundary maps (see §1.2). For $p\leq 2$ , we are able to prove the existence of well-suited boundary maps under geometric density (a hypothesis to which we can easily reduce). Unfortunately, for $p>2$ , we can only prove it under Zariski density, which is a stronger assumption.

1.2 Boundary maps and standard algebraic groups

To prove Theorem 1.1, we use, as it is now standard in bounded cohomology, boundary map techniques. Let ${\Gamma }$ be a lattice in $\operatorname {\mathrm{SU}}(1,n)$ and P a strict parabolic subgroup of $\operatorname {\mathrm{SU}}(1,n)$ . The space $B=\operatorname {\mathrm{SU}}(1,n)/P$ is a measurable ${\Gamma }$ -space which is amenable and has very strong ergodic properties, and is thus a strong boundary (see Definition 4.7) in the sense of [BF14]. This space can be identified with the visual boundary of the hyperbolic space ${\mathcal X}_{\mathbf{C}}(1,n)$ .

In finite dimension, for example in [Poz15], the target of the boundary map is the Shilov boundary of the symmetric space ${\mathcal X}_{\mathbf{C}}(p,q)$ . If $p\leq q$ , this Shilov boundary can be identified with the space $\mathcal I_p$ of isotropic linear subspaces of dimension p in ${\mathbf{C}}^{p+q}$ . In our infinite-dimensional setting, we use the same space $\mathcal I_p$ of isotropic linear subspaces of dimension p.

A main difficulty appears in this infinite-dimensional context: this space is not compact anymore for the natural Grassmann topology. Thus the existence of boundary $\Gamma $ -maps $B\to \mathcal I_p$ is more involved than in finite dimension. Such boundary maps have been obtained in a non-locally compact setting when the target is the visual boundary $\partial {\mathcal X}$ of a CAT(0) space ${\mathcal X}$ of finite telescopic dimension, on which a group $\Gamma $ acts isometrically [BDL16, Duc13]. Here, $\mathcal I_p$ is only a closed G-orbit of $\partial {\mathcal X}_{\mathbf{C}}(p,\infty )$ . Actually, $\mathcal I_p$ is a subset of the set of vertices in the spherical building structure on $\partial {\mathcal X}_{\mathbf{C}}(p,\infty )$ and the previous result is not sufficient. To prove the existence of boundary maps to $\mathcal I_p$ , we reduce to representations whose images are dense, in the sense that is explained below.

Following [CM09], we say that a group $\Gamma $ acting by isometries on a symmetric space (possibly of infinite dimension) of non-positive curvature is geometrically dense if there is no strict closed invariant totally geodesic subspace (possibly reduced to a point) or fixed point in the visual boundary. For finite-dimensional symmetric spaces, the geometric density is equivalent to Zariski density in the isometry group, which is a real algebraic group. To prove Theorem 1.7, we rely also on the theory of algebraic groups in infinite dimension introduced in [HK77]. Roughly speaking, a subgroup of the group of invertible elements of a Banach algebra is algebraic if it is defined by (possibly infinitely many) polynomial equations with a uniform bound on the degrees of the polynomials.

This notion of algebraic groups is too coarse for our goals and we introduce the notion of standard algebraic groups in infinite dimension. Let ${\mathcal H}$ be a Hilbert space and let $\operatorname {\textrm {GL}}({\mathcal H})$ be the group of invertible bounded operators of ${\mathcal H}$ . An algebraic subgroup of $\operatorname {\textrm {GL}}({\mathcal H})$ is standard if it is defined by polynomial equations in the matrix coefficients $g\mapsto \langle ge_i,e_j\rangle $ , where $(e_i)$ is some Hilbert base of ${\mathcal H}$ . See Definition 3.4. With this definition, we are able to show that stabilizers of points in $\partial X_{{\mathbf K}}(p,\infty )$ are standard algebraic subgroups, and the same holds for stabilizers of proper totally geodesic subspaces.

Definition 1.4. A subgroup of $\operatorname {{\textrm O_{\mathbf {K}}}}(p,\infty )$ is Zariski dense when it is not contained in a proper standard algebraic group. A representation $\rho \colon {\Gamma }\to \operatorname {\mathrm{PO}}_{{\mathbf K}}(p,\infty )$ is Zariski dense if the preimage of $\rho ({\Gamma })$ in $\operatorname {{\textrm O_{\mathbf {K}}}}(p,\infty )$ is Zariski dense. For a short discussion about a possible Zariski topology in infinite dimension, we refer to Remark 3.2.

We show in Proposition 1.5 that Zariski density implies geometric density.

Proposition 1.5. Let $p\in \mathbf{N}$ . Stabilizers of closed totally geodesic subspaces of ${\mathcal X}_{{\mathbf K}}(p,\infty )$ and stabilizers of points in $\partial {\mathcal X}_{{\mathbf K}}(p,\infty )$ are standard algebraic subgroups of $\operatorname {\mathrm{O}_{\mathbf {K}}}(p,\infty )$ .

In particular, a Zariski-dense subgroup of $\operatorname {\mathrm{O}_{\mathbf {K}}}(p,\infty )$ is geometrically dense.

Question 1.6. Is it true that the converse of Proposition 1.5 holds? Namely, are geometric density and Zariski density equivalent? It is also possible that one needs to strengthen the definition of standard algebraic groups to ensure that geometric density and Zariski density are the same.

Finally, we get the existence of the desired boundary maps under geometric or Zariski density. In the following statement, two linear subspaces are said to be transverse if their intersection is trivial.

Theorem 1.7. Let $\Gamma $ be a countable group with a strong boundary B and $p\in \mathbf{N}$ .

If $\Gamma $ acts geometrically densely on ${\mathcal X}_{{\mathbf K}}(p,\infty )$ with $p\leq 2$ , then there is a measurable $\Gamma $ -equivariant map $\phi \colon B\to \mathcal I_p$ . Moreover, for almost all pairs $(b,b')\in B^2$ , $\phi (b)$ and $\phi (b')$ are transverse.

If $\Gamma \to \operatorname {\mathrm{PO}}_{{\mathbf K}}(p,\infty )$ is a representation with a Zariski-dense image, then there is a measurable $\Gamma $ -equivariant map $\phi \colon B\to \mathcal I_p$ . Moreover, for almost all pairs $(b,b')\in B^2$ , $\phi (b)$ and $\phi (b')$ are transverse.

1.3 Geometry of chains

In [Car32], Cartan introduced a very nice geometry on the boundary $\partial {\mathcal X}_{\mathbf{C}}(1,n)$ of the complex hyperbolic space. A chain in $\partial {\mathcal X}_{\mathbf{C}}(1,n)$ is the boundary of a complex geodesic in ${\mathcal X}_{\mathbf{C}}(1,n)$ . It is an easy observation that any two distinct points in $\partial {\mathcal X}_{\mathbf{C}}(1,n)$ define a unique chain; moreover, to determine if three points lie in a common chain, one can use a numerical invariant, the so-called Cartan invariant. Three points lie in a common chain if and only if they maximize the absolute value of the Cartan invariant. This invariant can be understood as an angle or the oriented area of the associated ideal triangle. See [Gol99, §7.1].

As in [Poz15], we use a generalization of chains and of the Cartan invariant to prove our rigidity statements. For $p\geq 1$ and $q\in \mathbf{N}\cup \{\infty \}$ with $q\geq p$ , we denote by $\mathcal I_p(p,q)$ , or simply $\mathcal I_p$ if the pair $(p,q)$ is understood, the set of isotropic subspaces of dimension p in ${\mathbf{C}}^{p+q}$ . A p-chain (or simply a chain) in $\mathcal I_p(p,q)$ is the image of a standard embedding of $\mathcal I_p(p,p)$ in $\mathcal I_p(p,q)$ . This corresponds to the choice of a linear subspace of ${\mathbf{C}}^{p+q}$ where the Hermitian form has signature $(p,p)$ . A generalization of the Cartan invariant is realized by the Bergmann cocycle $\beta \colon \mathcal I_p^3\to [-p,p]$ . Two transverse points in $\mathcal I_p$ determine a unique chain and once again, three points in $\mathcal I_p^3$ that maximize the absolute value of the Bergmann cocycle lie in a common chain.

The strategy of proof of Theorem 1.1 goes now as follows. We first reduce to geometrically dense representations (Proposition 5.15) if needed. Thanks to a now well-established formula in bounded cohomology (Proposition 5.10), we prove that a maximal representation of a lattice $\Gamma \leq \operatorname {\mathrm{SU}}(1,n)$ in $\operatorname {\textrm O}_{\mathbf{C}}(p,\infty )$ has to preserve the chain geometry and almost surely maps 1-chains to p-chains (Corollary 6.1).

1.4 Outline of the paper

Section 2 is devoted to the background on Riemannian and Hermitian symmetric spaces in infinite dimension. Section 3 focuses on algebraic and standard algebraic subgroups where Proposition 1.5 is proved. The existence of boundary maps is proved in §4. In §5, we provide a short summary of the basic definitions related to maximal representations, and adapt them in infinite dimension. Section 6 deals with representations in $\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ , where we prove Theorem 1.1. In §7, we study representations of fundamental groups of surfaces in $\operatorname {\mathrm{PO}}_{\mathbf {R}}(2,\infty )$ and prove Theorem 1.2. The computation of the Toledo invariant for the variation on the complementary series is carried out in Appendix A.

2 Riemannian and Hermitian symmetric spaces of infinite dimension

2.1 Infinite-dimensional symmetric spaces

In this section, we recall definitions and facts about infinite-dimensional Riemannian symmetric spaces. By a Riemannian manifold, we mean a (possibly infinite-dimensional) smooth manifold modeled on some real Hilbert space with a smooth Riemannian metric. For a background on infinite-dimensional Riemannian manifolds, we refer to [Lan99] or [Pet06].

Remark 2.1. Implicitly, all Hilbert spaces considered in this paper will be separable. In particular, any two Hilbert spaces of infinite dimension over the same field will be isomorphic. The symmetric spaces studied below can be defined as well on non-separable Hilbert spaces but because we will consider representations of countable groups, we can always restrict ourselves to the separable case.

Let $(M,g)$ be a Riemannian manifold, a symmetry at a point $p\in M$ is an involutive isometry $\sigma _p\colon M\to M$ such that $\sigma _p(p) = p$ , and the differential at p is $-\operatorname {\textrm {Id}}$ . A Riemannian symmetric space is a connected Riemannian manifold such that, at each point, there exists a symmetry. See [Duc15a, §3] for more details.

We will be interested in infinite-dimensional analogs of symmetric spaces of non-compact type. If $(M,g)$ is a symmetric space of non-positive sectional curvature without local Euclidean factor, then for any point $p\in M$ , the exponential $\exp \colon T_pM\to M$ is a diffeomorphism and, if d is the distance associated to the metric g, then $(M,d)$ is a CAT(0) space [Duc15a, Proposition 4.1]. So, such a space M has a very pleasant metric geometry and in particular, it has a visual boundary $\partial M$ at infinity. If M is infinite-dimensional, then $\partial M$ is not compact for the cone topology.

Let us describe the principal example of an infinite-dimensional Riemannian symmetric space of non-positive curvature.

Example 2.2. Let ${\mathcal H}$ be some real Hilbert space with orthogonal group $\operatorname {\textrm {O}}(\mathcal {H})$ . We denote by $\operatorname {\mathrm{L}}({\mathcal H})$ the set of bounded operators on ${\mathcal H}$ and by $\operatorname {\textrm {GL}}({\mathcal H})$ the group of the invertible ones with continuous inverse. If $A\in \operatorname {\mathrm{L}}({\mathcal H})$ , we denote its adjoint by $^tA$ . An operator $A\in \operatorname {\mathrm{L}}({\mathcal H})$ is Hilbert–Schmidt if $\sum _{i,j}\langle Ae_i,Ae_j\rangle ^2<\infty $ , where $(e_i)$ is some orthonormal basis of ${\mathcal H}$ . We denote by $\operatorname {\mathrm{L}}^2({\mathcal H})$ the ideal of Hilbert–Schmidt operators and by $\operatorname {\textrm {GL}}^2({\mathcal H})$ the elements of $\operatorname {\textrm {GL}}({\mathcal H})$ that can be written $\operatorname {\textrm {Id}}+A$ , where $A\in \operatorname {\mathrm{L}}^2({\mathcal H})$ . This is a subgroup of $\operatorname {\textrm {GL}}({\mathcal H})$ : the inverse of $\operatorname {\textrm {Id}}+A$ is $\operatorname {\textrm {Id}}-B$ with $B=A(\operatorname {\textrm {Id}}+A)^{-1}=(\operatorname {\textrm {Id}}+A)^{-1}A\in \operatorname {\mathrm{L}}^2({\mathcal H})$ . We also set $\operatorname {\textrm {O}}^2(\mathcal {H})=\operatorname {\textrm {O}}(\mathcal {H})\cap \operatorname {\textrm {GL}}^2(\mathcal {H})$ , and denote by $\operatorname {\textrm {S}}^2(\mathcal {H})$ the closed subspace of symmetric operators in $\operatorname {\mathrm{L}}^2({\mathcal H})$ and by $\operatorname {\textrm {P}}^2(\mathcal {H})$ the set of symmetric positive definite operators in $\operatorname {\textrm {GL}}^2({\mathcal H})$ .

Then $\operatorname {\textrm {P}}^2(\mathcal {H})$ identifies with the quotient $\operatorname {\textrm {GL}}^2({\mathcal H})/\operatorname {\textrm {O}}^2(\mathcal {H})$ under the action of $\operatorname {\textrm {GL}}^2({\mathcal H})$ on $\operatorname {\textrm {P}}^2(\mathcal {H})$ given by $g\cdot x=gx^tg$ , where $g\in \operatorname {\textrm {GL}}^2({\mathcal H})$ and $x\in \operatorname {\textrm {P}}^2({\mathcal H})$ . The space $\operatorname {\textrm {P}}^2(\mathcal {H})$ is actually a Riemannian manifold, $\operatorname {\textrm {GL}}^2({\mathcal H})$ acts transitively by isometries, and the exponential map $\exp \colon \operatorname {\textrm {S}}^2(\mathcal {H})\to \operatorname {\textrm {P}}^2(\mathcal {H})$ is a diffeomorphism. The metric at the origin $o=\operatorname {\textrm {Id}}$ is given by $\langle X,Y\rangle =\text {Trace}(XY)$ and it has non-positive sectional curvature. Then it is a complete simply connected Riemannian manifold of non-positive sectional curvature. This is a Riemannian symmetric space and the symmetry at the origin is given by $x\mapsto x^{-1}$ .

A totally geodesic subspace of a Riemannian manifold $(M,g)$ is a closed submanifold N such that for any $x\in N$ and $v\in T_xN\setminus \{0\}$ , the whole geodesic with direction v is contained in N. All the simply connected non-positively curved symmetric spaces that will appear in this paper are totally geodesic subspaces of the space $\operatorname {\textrm {P}}^2(\mathcal {H})$ described in Example 2.2.

A Lie triple system of $\operatorname {\textrm {S}}^2(\mathcal {H})$ is a closed linear subspace $\mathfrak {p}<\operatorname {\textrm {S}}^2(\mathcal {H})$ such that for all $X,Y,Z\in \mathfrak {p}$ , $[X,[Y,Z]]\in \mathfrak {p}$ , where the Lie bracket $[X,Y]$ is simply $XY-YX$ . The totally geodesic subspaces N of $\operatorname {\textrm {P}}^2(\mathcal {H})$ containing $\operatorname {\textrm {Id}}$ are in bijection with the Lie triple systems $\mathfrak {p}$ of $\operatorname {\textrm {S}}^2(\mathcal {H})$ . This correspondence is given by $N=\exp (\mathfrak {p})$ . See [dlH72, Proposition III.4].

All totally geodesic subspaces of $\operatorname {\textrm {P}}^2(\mathcal {H})$ are symmetric spaces as well and satisfy a condition of non-positivity of the curvature operator. This condition of non-positivity of the curvature operator allows a classification of these symmetric spaces [Duc15a, Theorem 1.8]. In this classification, all the spaces that appear are the natural analogs of the classical finite-dimensional Riemannian symmetric spaces of non-compact type.

The isometry group of a finite-dimensional symmetric space is a real algebraic group and thus has a Zariski topology; this is no more available in infinite dimension. Let $(M,g)$ be an irreducible symmetric space of finite dimension and non-positive sectional curvature, and let $G\leq \operatorname {\mathrm{Isom}}(M)$ . It is well known that the group G is Zariski dense if and only if there is neither a fixed point at infinity nor an invariant totally geodesic strict subspace (possibly reduced to a point). Thus, following the ideas in [CM09], we say that a group G acting by isometries on a (possibly infinite-dimensional) Riemannian symmetric space of non-positive curvature ${\mathcal X}$ is geometrically dense if G has no fixed point in $\partial {\mathcal X}$ and no invariant closed totally geodesic strict subspace in ${\mathcal X}$ .

2.2 The Riemannian symmetric spaces ${\mathcal X}_{{\mathbf K}}(p,\infty )$

Throughout the paper, $\textbf {H}$ denotes the division algebra of the quaternions, and $\mathcal {H}$ is a separable Hilbert space over ${\textbf K}=\textbf {R}$ , ${\mathbf{C}}$ , or $\textbf {H}$ of infinite dimension. In the latter case, the scalar multiplication is understood to be on the right. We denote by $\operatorname {\mathrm{L}}(\mathcal {H})$ the algebra of all bounded ${\textbf K}$ -linear operators of $\mathcal {H}$ , and $\operatorname {\textrm {GL}}(\mathcal {H})$ is the group of all bounded invertible ${\textbf K}$ -linear operators with bounded inverse. Using the real Hilbert space $\mathcal {H}_{\mathbf {R}}$ underlying $\mathcal {H}$ , one can consider $\operatorname {\textrm {GL}}(\mathcal {H})$ as a closed subgroup of $\operatorname {\textrm {GL}}(\mathcal {H}_{\mathbf {R}})$ . We denote by $A^*$ the adjoint of $A\in \operatorname {\mathrm{L}}({\mathcal H})$ . In particular, when ${\textbf K}=\textbf {R}$ , $A^*= ^t\!A$ .

Let $p\in \mathbf{N}$ . We fix an orthonormal basis $(e_i)_{i\in \mathbf{N}}$ of the separable Hilbert space $\mathcal {H}$ , and we consider the Hermitian form

$$ \begin{align*}Q(x)=\sum_{i=1}^p\overline x_ix_i-\sum_{i>p}\overline x_ix_i,\end{align*} $$

where $x=\sum e_ix_i$ . The isometry group of this quadratic form will be denoted $\operatorname {{\textrm O_{\mathbf {K}}}}(Q)$ or equivalently $\operatorname {{\textrm O_{\mathbf {K}}}}(p,\infty )$ .

The intersection of $\operatorname {{\textrm O_{\mathbf {K}}}}(p,\infty )$ and the orthogonal group of $\mathcal {H}$ is isomorphic to $\operatorname {{\textrm O_{\mathbf {K}}}}(p)\times \operatorname {{\textrm O_{\mathbf {K}}}}(\infty )$ , where $\operatorname {{\textrm O_{\mathbf {K}}}}(p)$ (respectively $\operatorname {{\textrm O_{\mathbf {K}}}}(\infty )$ ) is the orthogonal group of the separable Hilbert space of dimension p (respectively of infinite dimension). Then the quotient

$$ \begin{align*}{\mathcal X}={\mathcal X}_{{\mathbf K}}(p,\infty)= \operatorname{{\textrm O_{\mathbf{K}}}}(p,\infty)/(\operatorname{{\textrm O_{\mathbf{K}}}}(p)\times\operatorname{{\textrm O_{\mathbf{K}}}}(\infty))\end{align*} $$

has a structure of an infinite-dimensional irreducible Riemannian symmetric space of non-positive curvature. This can be seen by the identification of ${\mathcal X}_{{\mathbf K}}(p,\infty )$ with the set

$$ \begin{align*}\mathcal{V}=\{V\leq{\mathcal H},\ \dim_{{\mathbf K}}(V)=p,\ Q|_V>0\}.\end{align*} $$

The group $\operatorname {{\textrm O_{\mathbf {K}}}}(p,\infty )$ acts transitively on $\mathcal {V}$ (by Witt’s theorem) and the stabilizer of the span of the p first vectors is $\operatorname {{\textrm O_{\mathbf {K}}}}(p)\times \operatorname {{\textrm O_{\mathbf {K}}}}(\infty )$ . Moreover, the subgroup $\operatorname {\textrm {O}}_{{\mathbf K}}^2(p,\infty )=\operatorname {{\textrm O_{\mathbf {K}}}}(p,\infty )\cap \operatorname {\textrm {GL}}^2({\mathcal H})$ also acts transitively on $\mathcal {V}$ and thus

$$ \begin{align*}{\mathcal X}_{{\mathbf K}}(p,\infty)\simeq \mathcal{V}\simeq \operatorname{\textrm{O}}_{{\mathbf K}}^2(p,\infty)/(\operatorname{{\textrm O_{\mathbf{K}}}}(p)\times\operatorname{\textrm{O}}_{{\mathbf K}}^2(\infty)).\end{align*} $$

The stabilizer of the origin in the action of $\operatorname {\textrm {O}}_{{\mathbf K}}^2(p,\infty )$ on $\operatorname {\textrm {P}}^2({\mathcal H}_{\mathbf {R}})$ is exactly $\operatorname {{\textrm O_{\mathbf {K}}}}(p)\times \operatorname {\textrm {O}}_{{\mathbf K}}^2(\infty )$ and the orbit of $\operatorname {\textrm {O}}_{{\mathbf K}}^2(p,\infty )$ in $\operatorname {\textrm {P}}^2({\mathcal H}_{\mathbf {R}})$ is a totally geodesic subspace [Duc13, Proposition 2.3]. Thus ${\mathcal X}_{{\mathbf K}}(p,\infty )$ has a structure of a simply connected non-positively curved Riemannian symmetric space.

Observe that when ${\textbf K}=\textbf {R}$ or ${\mathbf{C}}$ , homotheties act trivially on ${\mathcal X}_{{\mathbf K}}(p,\infty )$ and thus the group $\operatorname {\mathrm{PO}}_{{\mathbf K}}(p,\infty )$ , defined to be $\operatorname {\textrm {O}}_{{\mathbf K}}(p,\infty )/\{\unicode{x3bb} \operatorname {\textrm {Id}},\ |\unicode{x3bb} |=1\}$ , acts by isometries on ${\mathcal X}_{{\mathbf K}}(p,\infty )$ . Moreover, it is proved in [Duc13, Theorem 1.5] that this is exactly the isometry group of ${\mathcal X}_{{\mathbf K}}(p,\infty )$ when ${\textbf K}=\textbf {R}$ .

Definition 2.3. Let ${\mathcal X}_1,{\mathcal X}_2$ be two symmetric spaces of type ${\mathcal X}_{{\mathbf K}}(p_i,q_i)$ , where $p_i\leq q_i\in \mathbf{N}\cup \{\infty \}$ , corresponding to Hilbert spaces ${\mathcal H}_1,{\mathcal H}_2$ and Hermitian forms $Q_1,Q_2$ . By a standard embedding, we mean the data of a linear map $f\colon {\mathcal H}_1\to {\mathcal H}_2$ such that $Q_2(f(x),f(y))=Q_1(x,y)$ for all $x,y\in {\mathcal H}_1$ . The group $\operatorname {\textrm {O}}_{{\mathbf K}}(Q_1)$ embeds in $\operatorname {\textrm {O}}_{{\mathbf K}}(Q_2)$ in the following way: f intertwines the actions on ${\mathcal H}_1$ and $f({\mathcal H}_1)$ and the action is trivial on the orthogonal of $f({\mathcal H}_1)$ , which is a supplementary of $f({\mathcal H}_1)$ because $Q_2$ is non-degenerate on $f({\mathcal H}_1)$ .

Finally the totally geodesic embedding ${\mathcal X}_1\hookrightarrow {\mathcal X}_2$ is given by the orbit of the identity under the action of the orthogonal group of $Q_1$ .

The spaces ${\mathcal X}_{{\mathbf K}}(p,\infty )$ , with p finite, are very special among infinite-dimensional Riemannian symmetric spaces: they have finite rank, which is p. This means there are totally geodesic embeddings of $\textbf {R}^p$ in ${\mathcal X}_{{\mathbf K}}(p,\infty )$ but there are no totally geodesic embeddings of $\textbf {R}^q$ for $q>p$ . Furthermore, every infinite-dimensional irreducible Riemannian symmetric space of non-positive curvature operator and finite rank arises in this way [Duc15a].

This finite-rank property gives some compactness on $\overline {{\mathcal X}}={\mathcal X}\cup \partial {\mathcal X}$ for a weaker topology [CL10, Remark 1.2]. Moreover, we have the following important property.

Proposition 2.4. [Duc13, Proposition 2.6]

Any finite configuration of points, geodesics, points at infinity, flat subspaces of ${\mathcal X}_{{\mathbf K}}(p,\infty )$ is contained in some finite-dimensional totally geodesic subspace of ${\mathcal X}_{{\mathbf K}}(p,\infty )$ which is a standard embedding of ${\mathcal X}_{{\mathbf K}}(p,q)$ with $q\in \mathbf{N}$ .

The boundary at infinity $\partial {\mathcal X}_{{\mathbf K}}(p,\infty )$ has a structure of a spherical building, which we now recall. We refer to [AB08] for general definitions and facts about buildings, and to [Duc13] for the specific case in which we are interested. The space $\partial {\mathcal X}_{{\mathbf K}}(p,\infty )$ has a natural structure of a simplicial complex (of dimension $p-1$ ): a simplex (of dimension $r-1$ ) in $\partial {\mathcal X}_{{\mathbf K}}(p,\infty )$ is defined by a flag $(V_1\subsetneq \cdots \subsetneq V_r)$ , where all the $V_i$ are non-zero totally isotropic subspaces of ${\mathcal H}$ . In particular, vertices of $\partial {\mathcal X}_{{\mathbf K}}(p,\infty )$ correspond to totally isotropic subspaces. A simplex A is contained in a simplex B if all the subspaces appearing in the flag A also appear in the flag B.

Each vertex has a type, which is a number between $1$ and p given by the dimension of the corresponding isotropic subspace. More generally, the type of a cell is the finite increasing sequence of dimensions of the isotropic subspaces in the associated isotropic flag.

Definition 2.5. Two vertices of $\partial {\mathcal X}_{{\mathbf K}}(p,\infty )$ , corresponding to isotropic spaces V and W, are opposite if the restriction of Q to $V+W$ is non-degenerate, which means $W\cap V^{\bot }=0$ .

Two simplices of the same type, corresponding to two flags $(V_1\subset \cdots \subset V_r)$ and $(W_1\subset \cdots \subset W_r)$ of the same type, are opposite if their vertices of the same type are opposite.

Remark 2.6. In terms of CAT(0) geometry, we note that two vertices of $\partial {\mathcal X}_{{\mathbf K}}(p,\infty )$ of the same type are opposite if and only if they are joined by a geodesic line in ${\mathcal X}_{{\mathbf K}}(p,\infty )$ . For vertices of dimension p, opposition is equivalent to transversality: two vertices $V,W$ with $\dim (V)=\dim (W)=p$ are opposite if and only if $V\cap W=0$ .

2.3 Hermitian symmetric spaces

Let $(M,g)$ be a Riemannian manifold (possibly of infinite dimension). An almost complex structure is a $(1,1)$ -tensor J such that for any vector field X, $J(J(X))=-X$ . A triple $(M,g,J)$ , where $(M,g)$ is a Riemannian manifold and J is an almost complex structure, is a Hermitian manifold if for all vector fields $X,Y$ , $g(J(X),J(Y))=g(X,Y)$ . If $(M,g,J)$ is a Hermitian manifold, we define a 2-form $\omega $ by the formula $\omega (X,Y)=g(J(X),Y)$ . A Kähler manifold is a Hermitian manifold such that $d\omega =0$ and $\omega $ is the Kähler form on M.

Let $(M,g,J)$ be a Hermitian manifold and $\nabla $ be the Levi-Civita connection associated to the Riemannian metric g. The almost-complex structure J is parallel if $\nabla J=0$ , that is, if for all vector fields $X,Y$ , $\nabla _X(JY)=J(\nabla _XY)$ . This parallelism condition implies that $\omega $ is parallel as well, that is, for all vector fields $X,Y,Z$ , $(\nabla _X\omega )(Y,Z)=0$ . Because $d\omega (X,Y,Z)=(\nabla _X\omega )(Y,Z)-(\nabla _Y\omega )(X,Z)+(\nabla _Z\omega )(X,Y)$ , the parallelism condition $\nabla J=0$ implies that $\omega $ is closed.

Let N be the Nijenhuis (2,0)-tensor on M, that is, for all vector fields $X,Y$ ,

$$ \begin{align*}N(X,Y)=2([X,Y]-[J(X),J(Y)]-J[J(X),Y]-J[X,J(Y)]).\end{align*} $$

The parallelism of J implies that this tensor vanishes. An almost-complex structure J with vanishing Nijenhuis tensor is called a complex structure. Thus a parallel almost-complex structure is a complex structure.

Definition 2.7. Let $(M,g)$ be a simply connected Riemannian symmetric space of non-positive sectional curvature. The symmetric space M is said to be a Hermitian symmetric space if it admits a Hermitian almost-complex structure J that is invariant under symmetries. This means that for any $p,q\in M$ ,

$$ \begin{align*}d\sigma_p\circ J_{\sigma_p(q)}\circ d\sigma_p=J_q\end{align*} $$

on the tangent space $T_qM$ . One also says that the symmetries are holomorphic.

Let us recall some notation in $\operatorname {\textrm {P}}^2(\mathcal {H})$ . We denote by o the origin in $\operatorname {\textrm {P}}^2(\mathcal {H})$ , that is, the identity $\operatorname {\textrm {Id}}$ of ${\mathcal H}$ . The symmetry $\sigma _o$ at the origin is the map $x\mapsto x^{-1}$ . The action $\tau $ of $\operatorname {\textrm {GL}}^2({\mathcal H})$ on $\operatorname {\textrm {P}}^2({\mathcal H})$ is given by $\tau (g)(x)=gx^tg$ . In particular, one has the relation

$$ \begin{align*}\sigma_o\circ\tau(g)=\tau(g^{-1})\circ\sigma_o.\end{align*} $$

The exponential map $\exp \colon \operatorname {\mathrm{L}}^2({\mathcal H})\to \operatorname {\textrm {GL}}^2({\mathcal H})$ is a local diffeomorphism around the origin and it induces a diffeomorphism $\exp \colon \operatorname {\textrm {S}}^2({\mathcal H})\to \operatorname {\textrm {P}}^2({\mathcal H})$ . In particular, we identify the tangent space at the origin $T_o\operatorname {\textrm {P}}^2({\mathcal H})$ with the Hilbert space $\operatorname {\textrm {S}}^2({\mathcal H})$ . The isotropy group of the origin, that is, the fixator of o, is $\operatorname {\textrm {O}}^2({\mathcal H})$ . It acts also on $\operatorname {\textrm {S}}^2({\mathcal H})$ by $g\cdot v=gv^tg$ and one has $g\exp (v)^tg=\exp (gv^tg)$ for all $v\in \operatorname {\textrm {S}}^2({\mathcal H})$ and $g\in \operatorname {\textrm {O}}^2({\mathcal H})$ . If K is a subgroup of $\operatorname {\textrm {O}}^2({\mathcal H})$ , we denote by $K^*$ its image in the isometry group of $\operatorname {\textrm {S}}^2({\mathcal H})$ .

The following proposition is a mere extension of a classical statement in finite dimension to our infinite-dimensional setting (see for example [Hel01, Proposition VIII.4.2]). It can be proved with the same methods.

Proposition 2.8. Let $(M,g)$ be a totally geodesic subspace of the symmetric space $\operatorname {\textrm {P}}^2(\mathcal {H})$ containing o (the identity element) and corresponding to the Lie triple system $\mathfrak {p}$ . Let G be the connected component of the stabilizer of M in $\operatorname {\textrm {GL}}^2(\mathcal {H})$ and let K be the isotropy subgroup of o in G. Assume there is an operator $J_0\colon \mathfrak {p}\to \mathfrak {p}$ such that:

1. (1) $J_0^2=-\operatorname {\mathrm{Id}}$ ;

2. (2) $J_0$ is an isometry; and

3. (3) $J_0$ commutes with all elements of $K^*$ .

Then there is a unique G-invariant almost-complex structure J on M which coincides with $J_0$ on $T_oM$ . Moreover, J is Hermitian and parallel. Thus, $(M,g,J)$ is a Hermitian symmetric space and a Kähler manifold.

Remark 2.9. It is well known that a finite-dimensional manifold with a complex structure J is a complex manifold, that is, a manifold modeled on ${\mathbf{C}}^n$ with holomorphic transition maps. The same result does not hold in full generality for infinite-dimensional manifolds but in the case of real analytic Banach manifolds, the result still holds [Bel05, Theorem 7]. The Hermitian symmetric spaces we consider have a real analytic complex structure and thus are complex manifolds. Nonetheless, we will not need this result.

In the remainder of this section, we exhibit the complex structures J on two classes of Hermitian symmetric spaces we will use later in the paper. Thanks to Proposition 2.8, it suffices to find $J_0$ with the required properties. The complex structures we describe are all the natural analogs of the corresponding complex structures in finite dimension.

Below, we use orthogonal decompositions ${\mathcal H}=V\oplus W$ and block decompositions for elements of $\operatorname {\mathrm{L}}({\mathcal H})$ . When we write $g=\Big [\begin {smallmatrix} A &B\\ C & D \end {smallmatrix}\Big ]$ , this means that $A=\pi _V\circ g|_V\in \operatorname {\mathrm{L}}(V)$ , $B=\pi _V\circ g|_W\in \operatorname {\mathrm{L}}(W,V)$ , $C=\pi _W\circ g|_V\in \operatorname {\mathrm{L}}(V,W)$ , and $D=\pi _W\circ g|_W\in \operatorname {\mathrm{L}}(W)$ .

2.3.1 The Hermitian symmetric space ${\mathcal X}_{\mathbf{C}}(p,\infty )$

Let $\mathcal {H}$ be a complex Hilbert space of infinite dimension. We denote by ${\mathcal H}_{\mathbf {R}}$ the underlying real Hilbert space. Let $V,W$ be closed orthogonal complex subspaces of dimension $p\in \mathbf{N}\cup \{\infty \}$ and $\infty $ such that ${\mathcal H}=V\oplus W$ . Let $I_{p,\infty }=\operatorname {\textrm {Id}}_V\oplus -\operatorname {\textrm {Id}}_W$ . Thus,

$$ \begin{align*}\operatorname{\textrm{O}}^2_{\mathbf{C}}(p,\infty)=\{g\in\operatorname{\textrm{GL}}^2({\mathcal H}),\ g^*I_{p,\infty}g=I_{p,\infty}\}.\end{align*} $$

The symmetric space ${\mathcal X}_{\mathbf{C}}(p,\infty )$ is the $\operatorname {\textrm {O}}^2_{\mathbf{C}}(p,\infty )$ -orbit of the identity in $\operatorname {\textrm {P}}^2({\mathcal H}_{\mathbf {R}})$ , that is, the image under the exponential map of the Lie triple system

$$ \begin{align*}\mathfrak p=\left\{\left[\begin{array}{@{}ll}0&A\\A^*&0\end{array}\right],\ A\in\operatorname{\textrm{L}}^2(W,V)\right\}.\end{align*} $$

The complex structure is induced by the endomorphism $J_0$ of $\mathfrak p$ defined by $J_0\Big [\begin {smallmatrix}0&A\\A^*&0\end {smallmatrix}\Big ]=\Big [\begin {smallmatrix}0&iA\\-iA^*&0\end {smallmatrix}\Big ]$ . Because the stabilizer of $\operatorname {\textrm {Id}}_{{\mathcal H}_{\mathbf {R}}}$ in $\operatorname {\textrm {O}}_{\mathbf{C}}(p,\infty )$ is given by all the operators that can be expressed as $\Big [\begin {smallmatrix}P&0\\0&Q\end {smallmatrix}\Big ]$ with $P\in \operatorname {\textrm {O}}^2_{\mathbf{C}}(V)$ and $Q\in \operatorname {\textrm {O}}^2_{\mathbf{C}}(W)$ , $J_0$ satisfies the conditions of Proposition 2.8.

2.3.2 The Hermitian symmetric space ${\mathcal X}_{\mathbf {R}}(2,\infty )$

Let ${\mathcal H}$ be a real Hilbert space of infinite dimension. Let $V,W$ be closed orthogonal subspaces of dimension two and $\infty $ such that ${\mathcal H}=V\oplus W$ . Let $I_{2,\infty }=\operatorname {\textrm {Id}}_V\oplus -\operatorname {\textrm {Id}}_W$ . Thus,

$$ \begin{align*}\operatorname{\textrm{O}}^2_{\mathbf{R}}(2,\infty)=\{g\in\operatorname{\textrm{GL}}^2({\mathcal H}),\ ^tgI_{2,\infty}g=I_{2,\infty}\}.\end{align*} $$

The symmetric space ${\mathcal X}_{\mathbf {R}}(2,\infty )$ is the $\operatorname {\textrm {O}}^2_{\mathbf {R}}(2,\infty )$ -orbit of the identity in $\operatorname {\textrm {P}}^2({\mathcal H})$ , that is, the image under the exponential map of the Lie triple system

$$ \begin{align*}\mathfrak p=\left\{\left[\begin{array}{@{}ll}0&A\\{}^tA&0\end{array}\right],\ A\in\operatorname{\textrm{L}}(W,V)\right\}.\end{align*} $$

Fix some orthonormal basis $(e_1,e_2)$ of V and let $I=\Big [\begin {smallmatrix}0&-1\\1&0\end {smallmatrix}\Big ]$ . This element belongs to the group $\operatorname {\textrm {SO}}_{\mathbf {R}}(2)$ , which is commutative. The complex structure is defined by $J_0\Big [\begin {smallmatrix}0 &A \\ {}^t A &0 \end {smallmatrix}\Big ]=\Big [\begin {smallmatrix}0 &IA \\-^{t}(IA)&0\end {smallmatrix}\Big ]$ . Because the stabilizer of $\operatorname {\textrm {Id}}_{\mathcal H}$ in the identity component of $\operatorname {\textrm {O}}^2_{\mathbf {R}}(2,\infty )$ is given by operators of the form $\Big [\begin {smallmatrix}P &0\\0 &Q\end {smallmatrix}\Big ]$ , with $P\in \operatorname {\textrm {SO}}_{\mathbf {R}}(2)$ and $Q\in \operatorname {\textrm {O}}^2_{\mathbf {R}}(\infty )$ , let us denote by $\operatorname {\textrm {O}}^{+}_{\mathbf {R}}(2,\infty )$ the set of elements in $\operatorname {\textrm {O}}_{\mathbf {R}}(2,\infty )$ of the form $\Big [\begin {smallmatrix} A &B\\ C & D \end {smallmatrix}\Big ]$ , where A has positive determinant. So there is a $\operatorname {\textrm {O}}^{+}_{\mathbf {R}}(2,\infty )$ -invariant complex structure on ${\mathcal X}_{\mathbf {R}}(2,\infty )$ . Let us denote by $\operatorname {\mathrm{PO}}_{\mathbf {R}}^+(2,\infty )$ the image of $\operatorname {\textrm {O}}^{+}_{\mathbf {R}}(2,\infty )$ under the quotient map $\operatorname {\textrm {O}}_{\mathbf {R}}(2,\infty )\to \operatorname {\mathrm{PO}}_{\mathbf {R}}(2,\infty )$ .

2.4 Tube-type Hermitian symmetric spaces

In finite dimension, the class of Hermitian symmetric spaces splits into two classes: those of tube type and those that are not of tube type. This distinction is important for the approach we use to understand maximal representations. For a definition in finite dimension, we refer to [BIW09]. Let us briefly recall that if ${\mathcal X}$ is a finite-dimensional Hermitian symmetric space, a chain is the boundary (as a subset of the Shilov boundary of ${\mathcal X}$ ) of a maximal tube-type subspace. By definition, if ${\mathcal X}$ is of tube type, there is a unique maximal tube-type subspace: ${\mathcal X}$ itself. However, if ${\mathcal X}$ is not of tube type, chains lie in a unique $\operatorname {\mathrm{Isom}}({\mathcal X})$ -orbit and it yields a new incidence geometry: the chain geometry (see [Poz15, §3]). Let us give an ad hoc definition of tube-type Hermitian symmetric spaces in infinite dimension.

Definition 2.10. An irreducible Hermitian symmetric space is of tube type if there is a dense increasing union of tube-type finite-dimensional totally geodesic holomorphic Hermitian symmetric subspaces.

Lemma 2.11. The Hermitian symmetric spaces ${\mathcal X}_{\mathbf{C}}(\infty ,\infty )$ , ${\mathcal X}_{\mathbf {R}}(2,\infty )$ , $\operatorname {\mathrm{Sp}}^2({\mathcal H})/\operatorname {\mathrm{U}}^2({\mathcal H})$ , and the space $\operatorname {\mathrm{O}}^{*2}(\infty )/\operatorname {\mathrm{U}}^2(\infty )$ are of tube type.

The Hermitian symmetric space ${\mathcal X}_{\mathbf{C}}(p,\infty )$ with $p<\infty $ is not of tube type.

Proof. The four first cases are simply the closure of an increasing union of Hermitian totally geodesic holomorphic subspaces isomorphic to respectively ${\mathcal X}_{\mathbf{C}}(n,n),{\mathcal X}_{\mathbf {R}}(2,n), \operatorname {\textrm {Sp}}(2n)/\operatorname {\mathrm{U}}(n)$ , and $\text {SO}^{*}(4n)/\operatorname {\mathrm{U}}(2n)$ . All those spaces are of tube type.

For ${\mathcal X}_{\mathbf{C}}(p,\infty )$ with $p<\infty $ , we know that any finite-dimensional totally geodesic and holomorphic Hermitian symmetric subspace Y is contained in some standard copy of ${\mathcal X}_{\mathbf{C}}(p,q)$ for $q>p$ large enough. In particular, if Y is of tube type, then it lies in some standard copy of ${\mathcal X}_{\mathbf{C}}(p,p)$ and thus standard copies of ${\mathcal X}_{\mathbf{C}}(p,p)$ are maximal finite-dimensional Hermitian symmetric subspaces of tube type.

Remark 2.12. Among the infinite-dimensional Hermitian symmetric spaces of tube type, ${\mathcal X}_{\mathbf {R}}(2,\infty )$ is remarkable. This is the only one to be of tube type and of finite rank.

Remark 2.13. A theory of tube-type domains and Jordan algebras in infinite dimension has been developed. We refer to [KU77] and references for an entrance to this subject. We do not rely on this theory.

3 Algebraic groups in infinite dimension

3.1 Algebraic subgroups of bounded operators of Hilbert spaces

Algebraic subgroups of finite-dimensional linear Lie groups are well understood and equipped with a useful topology: the Zariski topology. In infinite dimension, some new and baffling phenomena may appear. For example, one-parameter subgroups may be non-continuous. In [HK77], Harris and Kaup introduced the notion of linear algebraic groups and showed that they behave nicely with respect to the exponential map. In particular, the exponential map is a local homeomorphism and any point sufficiently close to the origin lies in some continuous one-parameter subgroup.

Let $A,B$ be two real Banach algebras and let $G(A)$ be the set of all invertible elements of A. A map $f\colon A\to B$ is a homogeneous polynomial map of degree n if there is a continuous n-linear map $f_0\in \operatorname {\mathrm{L}}^n(A,B)$ such that for any $a\in A$ , $f(a)=f_0(a,\ldots ,a)$ . Now, a map $f\colon A\to B$ is polynomial if it is a finite sum of homogeneous polynomial maps. Its degree is the maximum of the degrees that appear in the sum.

Let ${\mathcal H}$ be a real Hilbert space. The Banach algebras we will use are $\operatorname {\mathrm{L}}({\mathcal H})$ endowed with the operator norm and the field of real numbers $\textbf {R}$ . The group of invertible elements in $\operatorname {\mathrm{L}}({\mathcal H})$ is $\operatorname {\textrm {GL}}({\mathcal H})$ .

Definition 3.1. A subgroup G of $G(A)$ is an algebraic subgroup if there is a constant n and a set $\mathcal {P}$ of polynomial maps of degrees at most n on $A\times A$ such that

$$ \begin{align*}G=\{g\in G(A);\ P(g,g^{-1})=0,\text{ for all } P\in \mathcal{P}\}.\end{align*} $$

Observe that $\mathcal {P}$ may be infinite but the degrees of its elements are uniformly bounded. The main result of [HK77] is the fact that an algebraic subgroup is a Banach Lie group (with respect to the norm topology) and that the exponential map gives a homeomorphism in a neighborhood of the identity.

Remark 3.2. In this context, one could define a generalized Zariski topology by choosing the smallest topology such that zeros of polynomial maps (or standard polynomial maps, see below) are closed. This topology behaves differently from the finite-dimensional case because the Noetherian property does not hold. We will not use any such topology.

Moreover, the intersection of an infinite number of algebraic subgroups has no reason to be an algebraic group. Degrees of the defining polynomials may be unbounded.

Examples 3.3.

1. (1) Let ${\mathcal H}$ be a Hilbert space of infinite dimension over ${\mathbf{C}}$ and let ${\mathcal H}_{\mathbf {R}}$ be the underlying real Hilbert space. Let I be the multiplication by the complex number i. Then I is an isometry of ${\mathcal H}_{\mathbf {R}}$ of order two and

   $$ \begin{align*}\operatorname{\textrm{GL}}({\mathcal H})=\{g\in\operatorname{\textrm{GL}}({\mathcal H}_{\mathbf{R}}), gI=Ig\}.\end{align*} $$
   Because the map $M\mapsto MI-IM$ is linear on $\operatorname {\mathrm{L}}({\mathcal H})$ , $\operatorname {\textrm {GL}}({\mathcal H})$ is an algebraic subgroup of $\operatorname {\textrm {GL}}({\mathcal H}_{\mathbf {R}})$ . Similarly, if $\textbf {H}$ is the field of quaternions and ${\mathcal H}$ is a Hilbert space over $\textbf {H}$ (with underlying real Hilbert space ${\mathcal H}_{\mathbf {R}}$ ), then $\operatorname {\textrm {GL}}({\mathcal H})$ is an algebraic subgroup of $\operatorname {\textrm {GL}}({\mathcal H}_{\mathbf {R}})$ .

2. (2) Let ${\mathcal H}$ be a Hilbert space of infinite dimension over ${\textbf K}$ and ${\mathcal H}=V\oplus W$ be an orthogonal splitting where V has dimension $p\in \mathbf{N}$ . Let $I_{p,\infty }$ be the linear map $\operatorname {\textrm {Id}}_V\oplus -\operatorname {\textrm {Id}}_W$ . By definition, the group $\operatorname {\textrm {O}}_{{\mathbf K}}(p,\infty )$ is

   $$ \begin{align*}\operatorname{\textrm{O}}_{{\mathbf K}}(p,\infty)=\{g\in\operatorname{\textrm{GL}}({\mathcal H}),g^*I_{p,\infty}g=I_{p,\infty}\}\end{align*} $$
   and because the map $(L,M)\mapsto L^*I_{p,\infty }M$ is bilinear on $\operatorname {\mathrm{L}}({\mathcal H}_{\mathbf {R}})\times \operatorname {\mathrm{L}}({\mathcal H}_{\mathbf {R}})$ , the group $\operatorname {\textrm {O}}_{{\mathbf K}}(p,\infty )$ is a (real) algebraic subgroup of $\operatorname {\textrm {GL}}({\mathcal H}_{\mathbf {R}})$ . This is a particular case of [HK77, Example 4].

In finite dimension, linear algebraic groups of $\operatorname {\textrm {GL}}_n(\textbf {R})$ are given by polynomial equations in matrix coefficients. We generalize this notion to subgroups of $\operatorname {\textrm {GL}}(\mathcal {H})$ .

Definition 3.4. Let ${\mathcal H}$ be a real Hilbert space. A matrix coefficient is a linear map $f\colon \operatorname {\mathrm{L}}({\mathcal H})\to \textbf {R}$ such that there are $x,y\in {\mathcal H}$ such that $f(L)=\langle Lx,y\rangle $ for any $L\in \operatorname {\mathrm{L}}({\mathcal H})$ . A homogeneous polynomial map P on $\operatorname {\mathrm{L}}({\mathcal H})\times \operatorname {\mathrm{L}}({\mathcal H})$ of degree d is standard if there is an orthonormal basis $(e_n)_{n\in \mathbf{N}}$ of ${\mathcal H}$ , non-negative integers $m,l$ such that $d=m+l$ , and families of real coefficients $(\unicode{x3bb} _i)_{i\in \mathbf{N}^{2m}}$ , $(\mu _j)_{j\in \mathbf{N}^{2l}}$ such that for all $(M,N)\in \operatorname {\mathrm{L}}({\mathcal H})\times \operatorname {\mathrm{L}}({\mathcal H})$ , $P(M,N)$ can be written as an absolutely convergent series

where $P_i(M)\!=\!\prod _{k\!=\!0}^{m-1}\langle Me_{i_{2k}},e_{i_{2k+1}}\rangle $ for $i\in \mathbf{N}^{2m}$ and similarly $P_j(N)\!=\!\prod _{k\!=\!0}^{l-1}\langle Ne_{i_{2k}},e_{i_{2k+1}}\rangle $ for $j\in \mathbf{N}^{2l}$ .

A polynomial map is standard if it is a finite sum of standard homogeneous polynomial maps. A subgroup of $\operatorname {\textrm {GL}}({\mathcal H})$ is a standard algebraic subgroup if it is an algebraic subgroup defined by a family of standard polynomials.

Examples 3.5.

1. (1) Any matrix coefficient is a standard homogeneous polynomial map of degree one. For $x,y\in {\mathcal H}$ and any orthonormal basis $(e_n)_{n\in \mathbf{N}}$ , we define $x_n=\langle x,e_n\rangle $ and similarly $y_n=\langle y,e_n\rangle $ for any $n\in \mathbf{N}$ . Then the matrix coefficient $P(M)=\langle Mx,y\rangle $ is given by the series

   $$ \begin{align*}\sum_{i,j\in\textbf{N}}x_iy_j\langle Me_i,e_j\rangle.\end{align*} $$
   For any finite subset $K\subset \mathbf{N}^2$ finite containing $\{1,\ldots ,n\}^2$ ,
   $$ \begin{align*}\bigg|\sum_{i,j\leq n}x_iy_j\langle Me_i,e_j\rangle-\langle Mx,y\rangle\bigg|\leq \|M\| (\Vert x\Vert \Vert \pi_n(y)-y\Vert+\Vert y\Vert \Vert \pi_n(x)-x\Vert ),\end{align*} $$
   where $\pi _n(x)$ is the projection on the space spanned by the n first vectors of the basis. This implies that the series is absolutely convergent (see [Cho69, VII-3-§8]).

2. (2) The group $\operatorname {\textrm {O}}_{{\mathbf K}}(p,\infty )$ is not only an algebraic group, it is also a standard algebraic group. Let $(e_i)_{i\in \mathbf{N}}$ be an orthonormal basis of ${\mathcal H}$ adapted to the decomposition ${\mathcal H}=V\oplus W$ as in Example 3.3. An element $g\in \operatorname {\textrm {GL}}({\mathcal H})$ is in $\operatorname {\textrm {O}}_{{\mathbf K}}(p,\infty )$ if and only if $\langle I_{p,\infty }ge_i,ge_j\rangle =\langle I_{p,\infty }e_i,e_j\rangle $ for any $i,j\in \mathbf{N}$ . Because $\langle I_{p,\infty }ge_i,ge_j\rangle =\sum _{k\in \mathbf{N}}\langle ge_i,I_{p,\infty }e_k\rangle \langle g e_j,e_k\rangle $ and the coefficient $\langle ge_i,I_{p,\infty }e_k\rangle $ is $\varepsilon _k\langle ge_i,e_k\rangle $ with $\varepsilon _k=-1$ for $m\leq p$ and $\varepsilon _k=1$ for $m\geq p$ , we see that $\operatorname {\textrm {O}}_{{\mathbf K}}(p,\infty )$ is a standard algebraic group.

3. (3) Let ${\mathcal H}$ be a real Hilbert space and $V<{\mathcal H}$ be a closed subspace, then $H=\operatorname {\textrm {Stab}}(V)$ is a standard algebraic subgroup of $\operatorname {\textrm {GL}}({\mathcal H})$ . Actually,

   $$ \begin{align*}H=\{g\in\operatorname{\textrm{GL}}({\mathcal H}),\ \langle gx,y\rangle=0,\text{ for all } x\in V,y\in V^{\bot}\},\end{align*} $$
   and thus is a standard algebraic subgroup of $\operatorname {\textrm {GL}}({\mathcal H})$ .It follows that stabilizers of simplices of the building at infinity of ${\mathcal X}_{{\mathbf K}}(p,\infty )$ are standard algebraic subgroups of $\operatorname {\textrm {O}}_{{\mathbf K}}(p,\infty )$ . Moreover, if $\xi $ is a point at infinity, its stabilizer coincides with the stabilizer of the minimal simplex that contains it. See [Duc13, Proposition 6.1] for details in the real case. The same argument works as well over ${\mathbf{C}}$ and $\textbf {H}$ . In particular, stabilizers of points at infinity are standard algebraic subgroups.

Let H be a standard algebraic subgroup of $\operatorname {\textrm {GL}}({\mathcal H})$ . If E is a finite-dimensional subspace of ${\mathcal H}$ , we denote by $H_E$ the subgroup of elements $g\in H$ such that $g(E)=E$ , $g|_{E^{\bot }}=\operatorname {\textrm {Id}}$ . We identify $H_E$ with an algebraic subgroup of $\operatorname {\textrm {GL}}(E)$ . By a strict algebraic group H, we mean that H is algebraic and $ H\neq \operatorname {\textrm {GL}}({\mathcal H})$ .

Lemma 3.6. If H is a strict standard algebraic subgroup, then there is a finite-dimensional subspace $E\subset {\mathcal H}$ such that $H_E$ is a strict algebraic subgroup of $\operatorname {\mathrm{GL}}(E)$ .

Proof. Let $\mathcal {P}$ be the family of standard polynomials defining the algebraic subgroup H. Let $P\in \mathcal {P}$ be a non-constant standard polynomial. Choose an orthonormal basis $(e_n)$ such that P can be written as an absolutely convergent series. For $n\in \mathbf{N}$ , let us set $E_n$ to be the space spanned by $(e_1,\ldots ,e_n)$ .

In particular for n large enough, the restriction of P to pairs $(g,g^{-1})$ is a non-constant polynomial map on $\operatorname {\textrm {GL}}(E_n)$ and thus defines a strict algebraic subset of $\operatorname {\textrm {GL}}(E_n)$ .

Remark 3.7. Not all polynomial maps are standard. The set of compact operators $\operatorname {\mathrm{L}}_c({\mathcal H})$ is closed in $\operatorname {\mathrm{L}}({\mathcal H})$ (it is the closure of the set of finite-rank operators) and by Hahn–Banach theorem, there is a non-trivial bounded linear form that vanishes on $\operatorname {\mathrm{L}}_c({\mathcal H})$ . This linear form is not standard because it vanishes on all finite-rank operators.

Remark 3.8. By Proposition 1.5 (proved in §3.2), Zariski density (Definition 1.4) implies geometric density and we do not know if the converse holds. Lemma 3.6 shows that algebraic subgroups can be tracked by considering finite-dimensional subspaces. So, one can think that phenomena similar to those in the finite-dimensional case happen and this is maybe a clue that the converse implication between geometric density and Zariski density holds. In particular, one can show that if H is a strict algebraic subgroup of $\operatorname {\textrm {O}}_{{\mathbf K}}(p,\infty )$ such that there is some finite-dimensional subspace E with $H_E\neq \{\operatorname {\textrm {Id}}\}$ , then H is not geometrically dense.

3.2 Exterior products

Let ${\mathcal H}_{{\mathbf K}}$ be a Hilbert space over ${\textbf K}$ with Hermitian form Q of signature $(p,\infty )$ . We denote by ${\mathcal H}$ the underlying real Hilbert space and by $(\cdot ,\cdot )$ the real quadratic form $\Re (Q)$ .

The exterior product ${\textstyle \bigwedge }^k\mathcal {H}$ has a natural structure of pre-Hilbert space and there is a continuous representation $\pi _k\colon \operatorname {\textrm {GL}}(\mathcal {H})\to \operatorname {\textrm {GL}}({\textstyle \bigwedge }^k\mathcal {H})$ given by the formula $\pi _k(g)(x_1\wedge \cdots \wedge x_k)=gx_1\wedge \cdots \wedge gx_k$ . An orthonormal basis of ${\textstyle \bigwedge }^k\mathcal {H}$ is given by $(e_{i_1}\wedge \cdots \wedge e_{i_k})_{i\in \mathcal {I}}$ , where $\mathcal {I}$ is the set of elements $i\in \mathbf{N}^k$ such that $i_1<\cdots <i_k$ and $(e_i)$ is an orthonormal basis of $\mathcal {H}$ . In other words, if $\langle \cdot ,\cdot \rangle $ is the scalar product on ${\mathcal H}$ , then the bilinear form applied to two vectors $x_1\wedge \cdots \wedge x_k$ and $y_1\wedge \cdots \wedge y_k$ is given by the Gram determinant $\det ( \langle x_i,y_j\rangle _{i,j=1..k})$ . As usual, the completion of ${\textstyle \bigwedge }^k\mathcal {H}$ is denoted $\overline {{\textstyle \bigwedge }^k\mathcal {H}}$ .

The space ${\textstyle \bigwedge }^k\mathcal {H}$ is also endowed with a quadratic form built from Q. One defines $(x_1\wedge \cdots \wedge x_k,y_1\wedge \cdots \wedge y_k)$ to be $\det ((x_i,y_j))$ . As soon as $k\geq 2$ , this quadratic form is non-degenerate of signature $(\infty ,\infty )$ and extends continuously to $\overline {{\textstyle \bigwedge }^k\mathcal {H}}$ . Moreover, $\pi _k(\operatorname {{\textrm O_{\mathbf {K}}}}(Q))$ preserves this quadratic form.

Lemma 3.9. Let $(e_i)_{i\in \mathcal I}$ be an orthonormal basis of $\mathcal {H}$ and let $v,w$ be vectors of ${\textstyle \bigwedge }^k \mathcal {H}$ . There are families $(\unicode{x3bb} _i)$ and $(\mu _i)$ such that for any $g\in \operatorname {\mathrm{GL}}(\mathcal {H})$ ,

is a standard polynomial in g.

Proof. If suffices to write $v=\sum _{i\in \mathcal {I}}\bigwedge _{l=1}^k e_{i_l}\unicode{x3bb} _i$ and $w=\sum _{j\in \mathcal {I}}\bigwedge _{l=1}^k e_{j_l}\mu _j$ and express the scalar product of ${\textstyle \bigwedge }^k\mathcal {H}$ in the basis $(e_{i_1}\wedge \cdots \wedge e_{i_k})_{i\in \mathcal {I}}$ . The sum is absolutely convergent because $(\unicode{x3bb} _i)$ and $(\mu _j)$ are Hilbert coordinates. Finally, $(ge_{i_l},e_{j_{\sigma (l)}})=\langle ge_{i_l}, I_{p,\infty }e_{j_{\sigma (l)}}\rangle $ and writing $I_{p,\infty }e_{j_{\sigma (l)}}$ in the Hilbert base, one recovers an absolutely convergent series of matrix coefficients in the Hilbert base $(e_i)$ .

Lemma 3.10. Let V be a non-trivial subspace of ${\textstyle \bigwedge }^k\mathcal {H}$ . The stabilizer of $\overline {V}$ in $\operatorname {{\mathrm{O}_{\mathbf {K}}}}(Q)$ is a standard algebraic subgroup.

Proof. If V is a non-trivial subspace of ${\textstyle \bigwedge }^k\mathcal {H}$ , one can choose an orthonormal basis $(v_i)_{i\in I}$ of ${\textstyle \bigwedge }^k\mathcal {H}$ such that the closure $\overline {V}$ in the Hilbert completion $\overline {{\textstyle \bigwedge }^k\mathcal {H}}$ is the closed span of $(v_i)_{i\in I_0}$ for some $I_0\subset I$ .

Let H be the subgroup of $\operatorname {{\textrm O_{\mathbf {K}}}}(Q)$ stabilizing $\overline {V}$ . Thus, by Lemma 3.9, g belongs to H if and only if, for all $ i\in I_0$ and $j\in I\setminus I_0$ , we have $(\pi _k(g)v_i,v_j)=0$ . Thus, H is the algebraic subgroup of $\operatorname {{\textrm O_{\mathbf {K}}}}(Q)$ defined by the family of polynomials $\mathcal {P}=\{P_{ij}\}$ where $P_{ij}(g)=(\pi _k(g)v_i,v_j)$ .

Proof of Proposition 1.5

We have seen in Example 3.5 that stabilizers of points at infinity are standard algebraic subgroups. Assume $\mathcal Y$ is a strict totally geodesic subspace of ${\mathcal X}$ . Without loss of generality, we assume that $o\in \mathcal Y$ and thus $\mathcal Y$ corresponds to some Lie triple system $\mathfrak {p}< \operatorname {\textrm {S}}^2({\mathcal H})$ . Let $\mathfrak {k}=\overline {[\mathfrak {p},\mathfrak {p}]}$ and $\mathfrak m$ be the Lie algebra $\mathfrak {k}\oplus \mathfrak {p}\leq \operatorname {\mathrm{L}}^2({\mathcal H})$ . Because $\mathfrak m$ is a Lie algebra, $G=\exp (\mathfrak m)$ is a subgroup of $\operatorname {\textrm {GL}}^2({\mathcal H})$ that is generated by transvections along geodesics in $\mathcal Y$ . In particular, for any $h\in \operatorname {{\textrm O_{\mathbf {K}}}}(p,\infty )$ , h normalizes G if and only if h preserves $\mathcal Y$ . Because $G=\exp (\mathfrak m)$ , h normalizes G if and only if it stabilizes $\mathfrak m$ under the adjoint action (that is, $\operatorname {\textrm {Ad}}(h)(\mathfrak m)=\mathfrak m$ , which means that for any $X\in \mathfrak m$ , $hXh^{-1}\in \mathfrak m$ ). Because $\mathfrak m$ is a closed subspace of $\operatorname {\mathrm{L}}^2({\mathcal H})$ , we have the splitting $\operatorname {\mathrm{L}}^2({\mathcal H})=\mathfrak m\oplus \mathfrak m^{\bot }$ . So, h stabilizes $\mathfrak m$ if and only if for any $X\in \mathfrak m$ and $Y\in \mathfrak m^{\bot }$ , $\langle hXh^{-1},Y\rangle =0$ , where $\langle \ ,\ \rangle $ is the Hilbert–Schmidt scalar product. Finally, because the map $(M,N)\mapsto \langle MXN,Y\rangle $ is bilinear on $\operatorname {\mathrm{L}}({\mathcal H})\times \operatorname {\mathrm{L}}({\mathcal H})$ , H is an algebraic subgroup of $\operatorname {{\textrm O_{\mathbf {K}}}}(p,\infty )$ . It remains to show that these bilinear maps are standard. Let $(e_n)$ be an orthonormal basis of ${\mathcal H}$ and let $E_{i,j}=e_i\otimes e_j^*$ be the associated orthonormal basis of $\operatorname {\mathrm{L}}^2({\mathcal H})$ , that is, $E_{i,j}(x)=\langle x,e_j\rangle e_i$ . Thus, let us write $X=\sum _{i,j}X_{i,j}E_{i,j}$ and $Y=\sum _{i,j}X_{i,j}E_{i,j}$ to obtain

$$ \begin{align*}\langle MXN,Y\rangle=\sum_{i,j,k,l}X_{i,j}Y_{k,l}\langle ME_{i,j}N,E_{k,l}\rangle,\end{align*} $$

where

$$ \begin{align*} \langle ME_{i,j}N,E_{k,l}\rangle&=\operatorname{\textrm{Trace}}((ME_{i,j}N)^*E_{k,l})\\ &=\sum_{n\in\textbf{N}}\langle N^*E_{i,j}M^*E_{k,l}(e_n),e_n\rangle\\ &=\langle N^*E_{j,i}M^*(e_k),e_l\rangle\\ &=\langle E_{j,i}M^*(e_k),N(e_l)\rangle\\ &=\langle e_i,M^*(e_k)\rangle\langle e_j,N(e_l)\rangle\\ &=\langle M(e_i),e_k\rangle\langle e_j,N(e_l)\rangle. \end{align*} $$

The absolute convergence of the series can be proven with the same arguments as in Example 3.5.(1).

Let ${\mathcal H}$ be a Hilbert space over ${\textbf K}$ with a non-degenerate Hermitian form Q of signature $(p,\infty )$ with $p\in \mathbf{N}$ . For a finite-dimensional non-degenerate subspace $E\subset {\mathcal H}$ of Witt index p, we denote by ${\mathcal X}_E\subset {\mathcal X}_{{\mathbf K}}(p,\infty )$ the subset of isotropic subspaces of E of dimension p. This corresponds to a standard embedding of ${\mathcal X}_{{\mathbf K}}(p,q)\hookrightarrow {\mathcal X}_{{\mathbf K}}(p,\infty )$ , where $(p,q)$ is the signature of the restriction of Q to E. Let $\mathcal {E}$ be the collection of all such finite-dimensional subspaces. We conclude this section with a lemma that shows that the family $({\mathcal X}_E)_{E\in \mathcal {E}}$ is cofinal among finite-dimensional totally geodesic subspaces.

Lemma 3.11. For any finite-dimensional totally geodesic subspace $\mathcal {Y}\subset {\mathcal X}_{{\mathbf K}}(p,\infty )$ , there is $E\in \mathcal {E}$ such that $\mathcal {Y}\subset {\mathcal X}_E$ .

Proof. We claim that one can find finitely many points $x_1,\ldots ,x_n\in \mathcal {Y}$ such that $\mathcal {Y}$ is the smallest totally geodesic subspace of ${\mathcal X}$ that contains $\{x_1,\ldots ,x_n\}$ . We define by induction points $x_1,\ldots ,x_k\in \mathcal {Y}$ and $\mathcal {Y}_k$ , that is, the smallest totally geodesic subspace containing $\{x_1,\ldots ,x_k\}$ . Observe that $\mathcal {Y}_k$ has finite dimension because $\{x_1,\ldots ,x_k\}\subset \mathcal {Y}$ and $\mathcal {Y}$ has finite dimension. For $x_1$ , choose any point in $\mathcal {Y}$ and let $\mathcal {Y}_1$ be $\{x_1\}$ . Assume $x_1,\ldots ,x_k$ have been defined. If $\mathcal {Y}_k\neq \mathcal {Y}$ , choose $x_{k+1}\in \mathcal {Y}\setminus \mathcal {Y}_k$ . One has $\mathcal {Y}_{k+1}\varsupsetneq \mathcal {Y}_{k}$ and thus $\dim (\mathcal {Y}_{k+1})>\dim (\mathcal {Y}_{k})$ . So, in finitely many steps, one gets that there is $n\in \mathbf{N}$ such that $\mathcal {Y}_n=\mathcal {Y}$ .

The points $x_1,\ldots ,x_n$ are positive definite subspaces (with respect to Q) of $\mathcal {H}$ . So, let E be the span of these subspaces. Observe that this space has finite dimension and $x_1,\ldots ,x_n\in {\mathcal X}_E$ . Up to adding finitely many vectors to E, we may moreover ensure that E is non-degenerate with Witt index p.

4 Boundary theory

4.1 Maps from strong boundaries

Let G be a locally compact, second countable group acting continuously by isometries on ${\mathcal X}_{{\mathbf K}}(p,\infty )$ , where p is finite. This section is dedicated to the analysis of Furstenberg maps also known as boundary maps from a measurable boundary of G to the geometric boundary $\partial {\mathcal X}_{{\mathbf K}}(p,\infty )$ , or more precisely, to some specific part of this boundary. We use a suitable notion of a measurable boundary of a group introduced in [BF14, §2]. This definition (see Definition 4.7) is a strengthening of previous versions introduced by Furstenberg [Fur73] and Burger and Monod [BM02].

Let us recall that a Polish space is a topological space which is separable and completely metrizable. By a Lebesgue G-space, we mean a standard Borel space (that is, a space and a $\sigma $ -algebra given by some Polish space and its Borel $\sigma $ -algebra), equipped with a Borel probability measure and an action of G which is measurable and preserves the class of the measure. We denote by $\textbf {P}(\Omega )$ the space of probability measures on a standard Borel space $\Omega $ . This is a Polish space for the topology of weak convergence.

Definition 4.1. Let $\Omega $ be a standard Borel space, $\unicode{x3bb} \in \textbf {P}(\Omega )$ . Assume that G acts on $\Omega $ with a measure-class preserving action. The action of G on $\Omega $ is isometrically ergodic if for every separable metric space Z equipped with an isometric action of G, every G-equivariant measurable map $\Omega \to Z$ is essentially constant.

Remark 4.2. If the action of G on $\Omega $ is isometrically ergodic, then it is ergodic (take $Z=\{0,1\}$ and the trivial action). Furthermore, if the action of G on $\Omega \times \Omega $ is isometrically ergodic, then it is also the case for the action on $\Omega $ .

Definition 4.3. Let Y and Z be two Borel G-spaces and $p:Y\to Z$ be a Borel G-equivariant map. We denote by $Y\times _p Y$ the fiber product over p, that is, the subset $\{(x,y)\in Y^2,\ p(x)=p(y)\}$ with its Borel structure coming from $Y^2$ .

We say that p (or Y) admits a fiberwise isometric action if there exists a Borel, G-invariant map $d:Y\times _p Y\to \textbf {R}$ such that any fiber $Y'\subset Y$ of p endowed with $d|_{Y'\times Y'}$ is a separable metric space.

Before going on, let us give a few examples of fiberwise isometric actions. These examples are closed to measurable fields of metric spaces that appear in [BDL16] and are simpler versions of fiberwise isometric actions that will appear in the proof of Theorem 1.7.

Example 4.4. Let $(M,d)$ be a metric space. The Wisjman hyperspace $2^M$ is the set of closed subspaces in M. This space can be embedded in the space $C(M)$ of continuous functions on M: to any close subspace A, one associates the distance function $x\mapsto d(A,x)$ . The topology of pointwise convergence on $C(M)$ induces the so-called Wisjman topology on $2^M$ . If $(M,d)$ is complete and separable, then the Wisjman hyperspace is a Polish space. Actually, when M is separable, the topology is the same as the topology of pointwise convergence on a countable dense subset.

Let ${\mathcal X}$ be a complete separable CAT(0) space. We denote by $\mathcal {F}_k$ the space of flat subspaces of dimension k in ${\mathcal X}$ (that is, isometric copies of $\textbf {R}^k$ ). One can check that $\mathcal {F}_k$ is closed in $2^{\cal X}$ : flatness is encoded in three conditions (equality in the CAT(0) inequality, convexity, and geodesic completeness), the dimension is encoded in the Jung inequality (see e.g. [LS97]), and all these conditions are closed. The visual boundary $\partial {\mathcal X}$ with the cone topology is a closed subspace of $\overline {{\mathcal X}}$ , which is an inverse limit of a countable family of closed balls [BH99]. Thus, $\partial {\mathcal X}$ is a Polish space.

Let G act by isometries on ${\mathcal X}$ and let $k>0$ be such that there exists a k-dimensional flat in ${\mathcal X}$ . Let $\partial \mathcal F_k=\{(F,\xi )\mid F\in \mathcal F_k\text { and } \xi \in \partial F\}$ . This is a closed subspace of $\mathcal {F}_k\times \partial {\mathcal X}$ . Then the continuous projection $\partial \mathcal F_k\to \mathcal F_k$ admits a fiberwise isometric action of $\operatorname {\mathrm{Isom}}(\cal X)$ , each $\partial F$ being endowed with the Tits metric.

Definition 4.5. Let A and B be Lebesgue G-spaces. Let $\pi :A\to B$ be a measurable G-equivariant map. We say that $\pi $ is relatively isometrically ergodic if each time we have a G-equivariant Borel map $p:Y\to Z$ of standard Borel G-spaces, which admits a fiberwise isometric action, and measurable G-maps $A\to Y$ and $B\to Z$ such that the following diagram commutes:

then there exists a measurable G-map $\phi \colon B\to Y$ which makes the following diagram commutative.

Remark 4.6. If a G-Lebesgue space B is such that the first projection $\pi _1\colon B\times B\to B$ is relatively isometrically ergodic, then it is isometrically ergodic. Indeed, if Y is a separable metric G-space and $f:B\to Y$ is G-equivariant, then it suffices to apply relatively isometric ergodicity to the map $\tilde {f}\colon (b,b')\mapsto f(b')$ and the trivial fibration $Y\to \{\ast \}$ .

Actually, relative isometric ergodicity yields a measurable map $\phi \colon B\to Y$ such that for almost all $(b,b')$ , $\phi (b)=f(b')$ , and thus f, is essentially constant.

Let B be a Lebesgue G-space. We use the definition of amenability for actions introduced by Zimmer, see [Zim84, §4.3]. The action $G\curvearrowright B$ is amenable if for any compact metrizable space M on which G acts continuously by homeomorphisms, there is a measurable G-equivariant map $\phi \colon B\to \textbf {P}(M)$ .

Definition 4.7. The Lebesgue G-space B is a strong boundary of G if:

• • the action of G on $(B,\nu )$ is amenable (in the sense of Zimmer); and

• • the first projection $\pi _1\colon B\times B\to B$ is relatively isometrically ergodic.

Example 4.8. The most important example for us is the following [BF14, Theorem 2.5]. Let G be a connected semisimple Lie group and P a minimal parabolic subgroup. Then $G/P$ , with the Lebesgue measure class, is a strong boundary for the action of G. If $\Gamma <G$ is a lattice, then $G/P$ is also a strong boundary for the action of $\Gamma $ . More generally, this is also true if G is a semisimple algebraic group over a local field.

The next example shows that every countable group admits a strong boundary.

Example 4.9. Let $\Gamma $ be a countable group, and $\mu \in \textbf {P}(\Gamma )$ be a symmetric measure whose support generates $\Gamma $ . Let $(B,\nu )$ be the Poisson-Furstenberg boundary associated to $(\Gamma ,\mu )$ . Then B is a strong boundary of $\Gamma $ [BF14, Theorem 2.7].

Existence of Furstenberg maps is already known. In the next section, we show that we can specify the type of points in the essential image and get that these points are essentially opposite. Let us recall that an isometric action of a group $\Gamma $ on a CAT(0) space ${\mathcal X}$ is non-elementary if there is neither invariant flat subspace (possibly reduced to a point) nor a global fixed point at infinity.

Theorem 4.10. [Duc13, Theorem 1.7]

Let ${\Gamma }$ be a locally compact second countable group, B a strong boundary for G, and $p\in \mathbf{N}$ . For any continuous and non-elementary action of ${\Gamma }$ on ${\mathcal X}_{{\mathbf K}}(p,\infty )$ , there exists a measurable ${\Gamma }$ -map $\phi : B \to \partial {\mathcal X}_{{\mathbf K}}(p,\infty ) $ .

We will also rely on results obtained in [BDL16]. Unfortunately, [BDL16] was written before the final version of [BF14] and a slightly different language was used there. Group actions on measurable metric fields were used there and here we just described fiberwise isometric actions. In the following proposition, we establish the relation between these two notions. We refer to [BDL16, §3] for definitions, notation, and a discussion about measurable metric fields. Roughly speaking, a measurable metric field over a Lebesgue $\Gamma $ -space $\Omega $ is a collection $\textbf {X}=(X_{\omega })_{\omega \in \Omega }$ of metric spaces $(X_{\omega },d_{\omega })$ where one moves from one such metric space to another one in a measurable fashion. It admits a $\Gamma $ -action if for all $\omega \in \Omega , g\in \Gamma $ , there is an isometry $\sigma (g,\omega )\colon X_{\omega }\to X_{g\omega }$ that satisfies the cocycle relation $\sigma (gg',\omega )=\sigma (g,g'\omega )\circ \sigma (g',\omega )$ almost surely.

Lemma 4.11. Let $\Gamma $ be a countable group and let $\mathrm{\mathbf{X}}$ be a measurable metric field over a Lebesgue $\Gamma $ -space $\Omega $ with a $\Gamma $ -action. Then there is a $\Gamma $ -invariant Borel subset $\Omega _0\subset \Omega $ of full measure, a standard Borel structure on $X=\bigsqcup _{\omega \in \Omega _0}X_{\omega }$ , and a Borel map $ p \colon X\to \Omega _0$ such that $ p $ admits a $\Gamma $ -fiberwise isometric action. Moreover, the fiber $ p ^{-1}(\omega )$ is $X_{\omega }$ with the metric $d_{\omega }$ .

If x is an invariant section of $\textbf {X}$ , then x corresponds canonically to a $\Gamma $ -equivariant measurable map $\Omega _0\to X$ .

Proof. Let $\{x^n\}_{n\in \mathbf{N}}$ be a fundamental family for the field $\textbf {X}$ . One can find a Borel subset $\Omega _0\subset \Omega $ of full measure such that all the maps $\omega \mapsto d_{\omega }(x_{\omega }^n,x^m_{\omega })$ are Borel for all $n,m\in \mathbf{N}$ on $\Omega _0$ . Observe that $\Omega _0$ is a Lebesgue space as well [Kec95, §12.B]. Up to replacing $\Omega _0$ by $\bigcap _{\gamma \in \Gamma }\gamma \Omega _0$ , we may assume that $\Omega _0$ is $\Gamma $ -invariant and still a Lebesgue space.

Let us set $X=\bigsqcup _{\omega \in \Omega _0}X_{\omega }$ and define $ p \colon X\to \Omega _0$ such that $ p (x)$ is the unique $\omega \in \Omega _0$ with $x\in X_{\omega }$ . Let us define $\phi _n\colon X\to \textbf {R}$ by the formula $\phi _n(x)= d_{ p (x)}(x,x^n_{ p (x)})$ . Now, let $\mathcal {A}$ be the smallest $\sigma $ -algebra such that $ p $ and $\phi _n$ are measurable for all n. To show that $(X,\mathcal {A})$ is a standard Borel space, it suffices to show that $\mathcal {A}$ is countably generated and separates points [Kec95, §12.B]. It is countably generated because $\Omega _0$ and $\textbf {R}$ are so. Let $x\neq y\in X$ . If $ p (x)\neq p (y)$ , then there is a Borel subset $\Omega '\subset \Omega _0$ such that $ p (x)\in \Omega '$ and $ p (y)\notin \Omega '$ thus $ p ^{-1}(\Omega ')\in \mathcal {A}$ separates x and y. If $ p (x)= p (y)=\omega $ , then by density of $\{x^n_{\omega }\}$ in $X_{\omega }$ , there is n such that $\phi _n(x)<\phi _n(y)$ , and thus $\mathcal {A}$ , separates x and y. Moreover, for $(x,y)\in X\times _ p X$ , we simply note $d(x,y)$ for $d_{p(x)}(x,y)$ . Then, $d(x,y)=\sup _{n\in \mathbf{N}}|\phi _n(x)-\phi _n(y)|$ and thus d is a Borel map. Because $\Omega _0$ is $\Gamma $ -invariant, $ p \colon X\to \Omega _0$ admits a fiberwise isometric $\Gamma $ -action.

If x is a section of $\textbf {X}$ , that is, an element of $\Pi _{\omega \in \Omega }X_{\omega }$ with measurability conditions [BDL16, Definitions 8 and 9], let us use the same notation for the map $x\colon \Omega _0\to X$ such that $x(\omega )=x_{\omega }$ . By construction, $\pi \circ x$ and $\phi _n\circ x$ are measurable and thus x is measurable. If the section is invariant, then this yields equivariance of the map $x\colon \Omega _0\to X$ .

Remark 4.12. With this lemma, for any two Lebesgue $\Gamma $ -spaces $A, B$ with a $\Gamma $ -factor map, that is, a measurable surjective $\Gamma $ -map $\pi \colon A\to B$ , the relative isometric ergodicity of $\pi $ , as stated in [BDL16, Definition 25], follows from Definition 4.5 above. Actually, if $\textbf {X}$ is a metric field over B, and $B_0$ , $p\colon X\to B_0$ are given by Lemma 4.11, this relative ergodicity is reflected in the following diagram where $A_0=\pi ^{-1}(B_0)$ .

This allows us to use freely the results from [BDL16].

4.2 Equivariant maps to the set of maximal isotropic subspaces under Zariski density

As before, let ${\mathcal H}$ be a Hilbert space over ${\textbf K}$ with a Hermitian form Q of signature $(p,q)$ with $p<q$ and $q\in \mathbf{N}\cup \{\infty \}$ . We fix some locally compact second countable group G with a continuous action by isometries on ${\mathcal X}_{{\mathbf K}}(p,q)$ and strong boundary B.

We denote by $\mathcal {I}_k$ the space of totally isotropic subspaces of $\mathcal {H}$ of dimension $k\leq p$ . Following the end of §2.2, this space can be identified with a type of vertices of the spherical building structure on $\partial {\mathcal X}_{p,q}$ . When recalling that the signature of the Hermitian form will seem to help comprehension, we will include it in our notation, and denote the space of totally isotropic subspaces as $\mathcal I_k(p,q)$ . Let us observe that if $p,q$ are finite, then $\mathcal I_k(p,q)$ can be identified with some homogeneous space $G/P$ , where $G=\operatorname {{\textrm O_{\mathbf {K}}}}(p,q)$ and P is a parabolic subgroup. In that case, we endow $G/P$ with the corresponding $\sigma $ -algebra and the unique G-invariant measure class on it (see e.g. [BdlHV08, Appendix B]). Observe that when $p=k=1$ (and q is finite or infinite), then $\mathcal I_1(1,q)$ is merely the visual boundary $\partial {\mathcal X}_{{\mathbf K}} (1,q)$ of the hyperbolic space ${\mathcal X}_{{\mathbf K}} (1,q)$ of dimension q over ${\textbf K}$ . For the application to maximal representations, the space $\mathcal {I}_p$ of totally isotropic subspaces of maximal dimension plays an important role.

For example, if ${\mathcal H}$ is a finite-dimensional complex vector space, ${\mathcal X}_{\mathbf{C}} (p,q)$ is a complex manifold admitting a bounded domain realization whose Shilov boundary can be $\operatorname {\mathrm{SU}}(p,q)$ -equivariantly identified with $\mathcal {I}_p$ . The purpose of this section is to associate to geometrically dense or Zariski-dense representations $\rho $ an equivariant boundary map with values in the set of maximal isotropic subspaces $\mathcal {I}_p$ (Theorem 1.7).

Remark 4.13. Under the hypothesis of Theorem 4.10, we get maps $B\to \mathcal {I}_k$ for at least one k: indeed, assume that $\phi $ is a map obtained by Theorem 4.10. Considering the smallest cell of the spherical building at infinity containing $\phi (b)$ , one gets a map $B\to \mathcal {F}$ , where $\mathcal {F}$ is a space of totally isotropic flags of $\mathcal {H}$ (see §6 in [Duc13]). Note that by ergodicity, the type of this flag is constant. Thus, for each dimension k that appears in this flag, one gets a map $B\to \mathcal {I}_k$ .

First, we prove opposition for boundary maps to $\mathcal {I}_k$ under Zariski density.

Proposition 4.14. Let $k\leq p$ and assume that ${\Gamma }$ is countable. Assume that the action ${\Gamma }\curvearrowright {\mathcal X}_{{\mathbf K}}(p,q)$ is Zariski dense. If $\phi \colon B\to \mathcal {I}_k$ is a ${\Gamma }$ -equivariant measurable map, then for almost every $(b,b')\in B\times B$ , $\phi (b)$ is opposite to $\phi (b')$ .

Proof. We denote by $V_b$ the linear subspace of dimension k corresponding to $\phi (b)$ and let $\ell _b$ be the corresponding line in ${\textstyle \bigwedge }^k\mathcal {H}$ . By ergodicity of the action ${\Gamma }\curvearrowright B\times B$ , one of the three following cases happens for almost all $(b,b')$ :

• • either $\ell _b=\ell _{b'}$ (which means that $V_b=V_{b'}$ );

• • $\ell _b$ and $\ell _{b'}$ span a totally isotropic plane in ${\textstyle \bigwedge }^k\mathcal {H}$ (in other words $0\neq V_b\cap V_{b'}^{\bot }\neq V_b$ );

• • or $\ell _b$ and $\ell _{b'}$ span a non-degenerate plane, that is, $V_b\cap V_{b'}^{\bot }=\{0\}$ . In other words, $V_b$ and $V_{b'}$ are opposite.

Our goal is to show that only the third case can happen. Assume first that $V_b=V_{b'}$ for almost every $(b,b')$ . Then the map $b\mapsto V_b$ is essentially constant and its essential image is a ${\Gamma }$ -invariant vertex. This contradicts the assumption that ${\Gamma }$ does not fix a point in $\partial {\mathcal X}_{{\mathbf K}}(p,\infty )$ .

Now assume that for almost every $(b,b')$ , the lines $\ell _b$ and $\ell _{b'}$ are orthogonal, namely their span is an isotropic plane. Then, thanks to Fubini’s theorem, there exists $b\in B$ and $B_b\subset B$ with full measure such that, for any $b'\in B_b$ , $\ell _{b'}$ is orthogonal to $\ell _b$ . Let $B'$ be the intersection $\bigcap _{\gamma \in {\Gamma }}\gamma B_b$ . The set $B'$ has full measure and is ${\Gamma }$ -invariant, thus the space spanned by $\{\ell _{b'},\ b'\in B'\}$ is a proper subspace (being included in the orthogonal of $\ell _b$ ) and is ${\Gamma }$ -invariant. The closure of this space is not $\operatorname {{\textrm O_{\mathbf {K}}}}(p,\infty )$ -invariant because this group acts transitively on the space of totally isotropic subspaces of dimension k. We conclude by using Lemma 3.10.

In the remainder of this section, our goal is to show the existence of maps from a strong boundary B to $\mathcal {I}_p$ . We begin our discussion by observing that for every k, there is a natural fiberwise isometric action of ${\Gamma }$ over $\mathcal {I}_k$ : we denote by $\mathcal V_k$ the space of subspaces V of $\mathcal {H}$ with dimension p such that $Q|_V$ is non-negative and $\ker (Q|_V)$ has dimension k. We endow both $\mathcal I_k$ and $\mathcal V_k$ with the induced topologies coming from the corresponding Grassmannians $\mathcal {G}_k,\mathcal {G}_p$ of subspaces of dimension k and p in $\mathcal {H}$ . Let us recall that a complete and separable distance on the Grassmannian $\mathcal {G}_m$ of all subspaces of dimension m is given by

$$ \begin{align*}d(V,W)^2=\sum_{i=1}^m\alpha_i^2,\end{align*} $$

where $\alpha _1,\ldots ,\alpha _m$ are the principal angles between V and $W\in \mathcal {G}_m$ . This topology also coincides with the Wisjman topology of the hyperspace $2^{\mathcal H}$ .

The natural projection

$$ \begin{align*}\begin{array}{@{}lll} \pi\colon \mathcal V_k&\to& \mathcal{I}_k\\ V&\mapsto &\ker(Q|_V) \end{array}\end{align*} $$

is continuous. For $V_0\in \mathcal {I}_k$ , the fiber $\pi ^{-1}(V_0)$ can be identified with a symmetric space ${\mathcal X}(V_0^{\bot }/V_0,Q)$ that we define in the following lines. The kernel of Q restricted to $V_0^{\bot }$ is exactly $V_0$ and thus Q defines a strongly non-degenerate Hermitian form on $V_0^{\bot }/V_0$ of signature $(p-k,q)$ . So, we define ${\mathcal X}(V_0^{\bot }/V_0,Q)$ to be the symmetric space associated to that Hermitian form, that is, the collection of positive subspaces of $V_0^{\bot }/V_0$ of dimension $p-k$ . The metric on ${\mathcal X}(V_0^{\bot }/V_0,Q)$ is given by the hyperbolic principal angles [Duc13, §3.1]. The preimages of such a positive subspace under the projection $V_0^{\bot }\to V_0^{\bot }/V_0$ are in bijective correspondence with the elements in the fiber of $\mathcal V_k$ above $V_0$ .

Recall that $V_0,W_0\in \mathcal {I}_k$ are opposite if the restriction of Q to $V_0+W_0$ is non-degenerate and thus has signature $(k,k)$ . If $V_0,W_0$ are opposite, then $\mathcal {H}=V_0\oplus W_0^{\bot }$ because $W_0^{\bot }$ has codimension k and $V_0\cap W_0^{\bot }=\{0\}$ . So, there is a bijective correspondence $\sigma _{V_0,W_0}\colon {\mathcal X}(V_0^{\bot }/V_0,Q)\to {\mathcal X}(W_0^{\bot }/W_0,Q)$ given by the formula

$$ \begin{align*}\sigma_{V_0,W_0}(V)=(V\cap W_0^{\bot})+W_0\end{align*} $$

for $V\in \pi ^{-1}(V_0)$ . This map is well defined because $V\cap W_0^{\bot }$ has dimension $p-k$ and is positive definite for Q. The inverse is given by

$$ \begin{align*}\sigma^{-1}_{V_0,W_0}(W)=\sigma_{W_0,V_0}(W)=(W\cap V_0^{\bot})+V_0.\end{align*} $$

Lemma 4.15. If $V_0,W_0\in \mathcal {I}_k$ are opposite, then the map $\sigma _{V_0,W_0}\colon {\mathcal X}(V_0^{\bot }/V_0,Q)\to {\mathcal X}(W_0^{\bot }/W_0,Q)$ is an isometry.

Proof. Because $V_0$ and $W_0$ are opposite, we have the following orthogonal decomposition:

$$ \begin{align*}\mathcal{H}=(V_0\oplus W_0)\oplus^{\bot}(V_0^{\bot}\cap W_0^{\bot}),\end{align*} $$

and the restriction of Q to $V_0^{\bot }\cap W_0^{\bot }$ is non-degenerate of signature $(p-k,\infty )$ . In particular, $V_0^{\bot }=V_0\oplus (V_0^{\bot }\cap W_0^{\bot })$ and thus the quotient map induces an isomorphism $(V_0^{\bot }/V_0,Q)\simeq (V_0^{\bot }\cap W_0^{\bot },Q)$ as spaces with Hermitian forms.

Now, if $V\in \pi ^{-1}(V_0)$ is written $V=V_0+V'$ , where $V'=V\cap W_0^{\bot }$ , then $V'\subset V_0^{\bot }\cap W_0^{\bot }$ . In particular, $\sigma _{W_0,V_0}(V)=V'+W_0$ . By construction of the metric through hyperbolic principal angles, the following map are isometries:

Finally, $\sigma _{W_0,V_0}$ is an isometry being the composition of two isometries.

In particular, $\sigma _{V_0,W_0}$ maps flat subspaces to flat subspaces.

Proof of Theorem 1.7 in the Zariski-dense case

In this proof, we freely use measurable metric fields thanks to Lemma 4.11. We know from Theorem 4.10 that there exists a ${\Gamma }$ -map to the visual boundary $\partial {\mathcal X}(p,\infty )$ and, by ergodicity, we get a ${\Gamma }$ -equivariant map $\phi \colon B\to \mathcal {I}_k$ for some $k\geq 1$ . Assume that k is a maximal such integer. If $k=p$ , we are done. Assume then that $k<p$ . Denoting $V_b\in \mathcal {I}_k$ for $\phi (b)$ and $X_b={\mathcal X}(V_b^{\bot }/V_b,Q)$ , we obtain a measurable field of (non-trivial) CAT(0) spaces $\textbf {X}=\{X_b\}$ with a $\Gamma $ -isometric action. Thanks to [Duc13, Theorem 1.8], either there is an invariant section of the metric field ${\partial {\textbf X}}=\{\partial X_b\}$ or there is a $\Gamma $ -equivariant Euclidean subfield $\textbf {F}=\{F_b\}$ with $F_b\subset {X}_b$ for almost all $b\in B$ .

In the first case, the stated result easily follows: to every point in $\partial X_b$ , one can associate a (non-trivial) totally isotropic flag in $V_b^{\bot }/V_b$ , and we can choose the totally isotropic subspace of maximal dimension in such a flag (whose dimension is essentially constant by ergodicity) to define a totally isotropic subspace $V^\prime _b$ of $\mathcal {H}$ strictly containing $V_b$ . Thus, we get a $\Gamma $ -map $\phi ^\prime \colon B\to \mathcal {I}_{k^\prime }$ for $k^\prime>k$ , which contradicts the maximality of k.

To conclude the proof, it is enough to show that, under our hypotheses, there cannot exist a $\Gamma $ -equivariant Euclidean subfield. So let us assume that there exists such a subfield. In other words, we have a $\Gamma $ -map $\psi _0\colon B\to F$ , where F is the Polish space constructed from $\textbf {F}$ thanks to Lemma 4.11, such that $\psi _0(b)$ is a flat in $X_b$ . Let us merely denote $F_b$ for $\psi _0(b)$ . Note that the map $b\mapsto \dim (F_b)$ is measurable [BDL16, Lemma 14] and hence $F_b$ is essentially of constant dimension. Among all possible such maps $\psi _0$ , we choose one such that this dimension, say $k_0$ , is minimal.

Let us denote $\sigma _{b,b^\prime }=\sigma _{\phi (b),\phi (b')}$ the correspondence isometry from $ X_b$ to $ X_{b^\prime }$ defined above.

We claim first that $\sigma _{b',b}(F_{b^\prime })$ is parallel to $F_b$ . The proof of this statement is very similar to that of [BDL16, Theorem 34]. We will explain the proof quickly and refer to [BDL16] for more details (in particular, about measurability of the various maps which appear during the proof).

Consider the function $f_{b,b'}$ defined on $F_b$ by $f_{b,b'}(x)=d(x,\sigma _{b',b}(F_{b^\prime }))$ (recall that both $F_b$ and $\sigma _{b',b}(F_{b^\prime })$ are flat subspaces of $X_b$ ). Then, $f_{b,b'}$ is a convex function on the Euclidean space $F_b$ . Using proposition 4 from [BDL16], we see that four cases are possible for $f_{b,b'}$ , which are described below. By the arguments from the proof of [BDL16, Theorem 34], these four conditions are measurable and ${\Gamma }$ -invariant, so that one of them happens almost surely.

The first case is when $f_{b,b'}$ does not attain its infimum m. In that case, one can consider the sequence $E_n^{b,b'}$ of a subset of $F_b$ defined as $E_n^{b,b'}=\{x\mid f(x)\leq m+1/n\}$ . By [Duc13, Proposition 8.10], this sequence of subsets gives a $\Gamma $ -map $\xi :B\times B\to \partial F$ , where $\partial F$ is the Borel space associated to the metric field $\partial \textbf {F}$ such that $\xi (b,b')\in \partial F_b$ for almost all $(b,b')$ . Because $\partial \textbf {F}$ is a metric field with a $\Gamma $ -action, using relative isometric ergodicity, we see that $\xi $ does not in fact depend on $b'$ , and therefore we have a map $\xi : B\to \partial X$ (where $\partial X$ is the Borel space associated to the metric field $\partial \textbf X$ ) such that $\xi (b)\in \partial X_b$ . Now $X_b={\mathcal X}(V_b^{\bot }/V_b,Q)$ has a boundary, which is a spherical building where cells correspond to totally isotropic flags in $V_b^{\bot }/V_b$ . Therefore, to a point in the boundary, one can associate a totally isotropic subspace $W\subset V_b^{\bot }/V_b$ , which we can lift to a totally isotropic space $\overline {W}$ containing $V_b$ in ${\mathcal H}$ . Thus the map $\xi $ gives rise to a map $B\to \mathcal I_{k'}$ with $k'>k$ , which contradicts the assumption on k.

If $f_{b,b'}$ attains its minimum m, let $Y=f_{b,b'}^{-1}(m)$ , which depends on b and $b'$ . The second case is when Y is bounded. Then one can consider its circumcenter $y(b,b')$ . The map $(b,b')\mapsto y(b,b')$ is measurable [Duc13, Lemma 8.7] and $\Gamma $ -equivariant. By relative isometric ergodicity, it does not in fact depend on $b'$ . So there is a $\Gamma $ -map $x\colon B\to X$ such that $x(b)\in X_b$ . In particular, $\{x(b)\}$ is a Euclidean subfield of $X_b$ and by minimality of $k_0$ , $F_b=\{x_b\}$ and thus we get parallelism of $\sigma _{b,b'}(F_{b'})$ and $F_b$ because any two points are parallel as Euclidean subspaces of dimension zero.

In the third case, one can write $Y=E\times T$ , where E is subflat and T is bounded. Let t be the circumcenter of T and let $E'=E\times \{t\}$ . Then $E'$ is a subflat of some dimension d which, by ergodicity, does not depend on $(b,b')$ . Now the set of subspheres of dimension d of a $k_0$ -dimensional Euclidean space is a metric field with a $\Gamma $ -invariant metric [BDL16, Lemma 20]. By relative isometric ergodicity, the map $(b,b')\mapsto \partial E'$ does not depend on $b'$ . By the second part of [BDL16, Lemma 20], the set of Euclidean subsets of $X_b$ , whose boundary is $\partial E'$ , is again a metric field with a $\Gamma $ -invariant metric. Thus, relative isometric ergodicity again allows us to conclude that the map $(b,b')\mapsto E'$ does not depend on $b'$ . In other words, we get a map which associates to b a subflat of $F_b$ . Because we assumed the dimension of $F_b$ to be minimal, this map must be equal to $\psi _0$ . This means that $f_{b,b'}$ is constant on $F_b$ , and therefore $F_b$ and $\sigma _{b',b}(F_b^\prime )$ are parallel.

In the last case, one can write $Y=E\times T$ , where T is unbounded, but $\partial T$ has a center. Then we get a map which associates to $(b,b')$ the center of $\partial T$ , which is a point in $\partial X_b$ . We conclude by the same argument as in the first case.

This concludes the proof of the claim: $F_b$ is (almost surely) parallel to $\sigma _{b',b}(F_{b'})$ . The set of flats parallel to $F_b$ is a metric field with a $\Gamma $ -invariant metric. Therefore, one can apply again relative isometric ergodicity to prove that the map $(b,b')\mapsto \sigma _{b',b}(F_{b'})$ does not depend on $b'$ . In other words, we get a map $\psi _1$ such that for almost all $b^\prime $ , $\sigma _{b,b^\prime }(\psi _0(b^\prime ))=\psi _1(b)$ . Let us denote $G_b=\psi _1(b)$ . One has $\sigma _{b,b'}(F_{b'})=G_b$ and because $\sigma _{b',b}=\sigma _{b,b'}^{-1}$ , one also has $\sigma _{b,b'}(G_{b'})=F_{b}$ . Thus, $\sigma _{b,b'}$ maps the flat equidistant to $F_{b'}$ and $G_{b'}$ maps the flat equidistant to $F_b$ and $G_b$ .

Up to replacing $\psi _0(b)$ by the flat equidistant to $\psi _0(b)$ and $\psi _1(b)$ , we may assume that $\psi _0(b)=\psi _1(b)$ and thus $\sigma _{b,b^\prime }(F_{b^\prime })=F_b$ for almost all $(b,b^\prime )\in B\times B$ . Let us recall that points in $X_b$ are positive definite subspaces $W\subset V_b^{\bot }/V_b$ and we denote by $\overline {W}$ the preimage of W under the quotient map $V_b^{\bot }\to V_b^{\bot }/V_b$ . Let us denote $\psi (b)=\overline {F_b}\subset \mathcal {V}_k$ , where $\overline {F_b}$ is the collection of $\overline {W}$ for $W\in F_b$ . Because $\sigma _{b,b^\prime }(F_{b^\prime })=F_b$ , if $W\in F_b$ , then $W'=\sigma _{b',b}(W)\in F_{b'}$ satisfies $\overline {W}=V_b+(\overline {W}\cap \overline {W'})$ . In particular, $\operatorname {\textrm {Span}}(\psi (b))\cap \operatorname {\textrm {Span}}(\psi (b'))\neq \{0\}$ for almost all $(b,b')$ . Moreover, the dimension of this intersection is essentially constant by ergodicity.

So, there is $b_0\in B$ such that for almost every b, $\operatorname {\textrm {Span}}(\psi (b_0))\cap \operatorname {\textrm {Span}}(\psi (b))\neq \{0\}$ and this set, $B^\prime $ , of full measure can be assumed to be $\Gamma $ -invariant. Let $\ell _b$ be the line corresponding to $\operatorname {\textrm {Span}}(\psi (b))$ in $\Lambda ^d\mathcal {H}$ , where d is the dimension of $\operatorname {\textrm {Span}}(\psi (b))$ . In particular, for all $b\in B'$ , $\ell _b$ is in the kernel of the map

$$ \begin{align*}\begin{array}{@{}lll} \Lambda^d\mathcal{H}&\to&\Lambda^{2d}\mathcal{H}\\ v&\mapsto&v\wedge v_{b_0}, \end{array}\end{align*} $$

where $v_{b_0}$ is a fixed non-trivial vector in $\ell _{b_0}$ . As in the proof of Proposition 4.14, the closure of the span of $\{\psi (b)\}_{b\in B'}$ is a non-trivial $\Gamma $ -invariant subspace in $\Lambda ^d\mathcal {H}$ . Thanks to Lemma 3.10, we have a contradiction with the Zariski-density assumption.

The statement about transversality is a direct consequence of Proposition 4.14.

4.3 Low-rank cases

In this subsection, we prove that if the rank of the target is at most two, then Zariski density can be relaxed to geometric density to obtain the desired boundary map. The difference between the rank-1 or -2 cases and the general case comes from the complexity of the relative positions of finitely many points in $\mathcal I_k$ for $k\leq p$ . This complexity increases with p but remains manageable in small ranks.

Theorem 4.16. Let $\Gamma $ be a countable group with strong boundary B and let $\rho \colon {\Gamma }\to \operatorname {\mathrm{PO}}_{{\mathbf K}}(p,\infty )$ be a representation. Assume that $p\leq 2$ and $\rho $ has no invariant linear subspace of dimension at most four. Then there is a $\Gamma $ -map $\phi \colon B\to \mathcal I_p$ such that for almost every $(b,b')\in B\times B$ , $\phi (b)$ is opposite to $\phi (b')$ .

Proof. Let us prove first that the induced action on ${\mathcal X}_{{\mathbf K}}(p,\infty )$ is non-elementary. If there is a fixed point at infinity, then there is an invariant isotropic subspace of dimension at most p and if there is a flat subspace of dimension d, the span of its points is a subspace of dimension $2d\leq 4$ .

In the case $p=1$ , the whole visual boundary is identified with $\mathcal I_p$ and opposition simply means that the map is not essentially constant (which is the case, otherwise there would be an invariant isotropic line). So the existence is guaranteed by Theorem 4.10 and it is not constant because there is no fixed point at infinity.

Now assume $p=2$ . We know the existence of a $\Gamma $ -map $\phi \colon B\to \mathcal I_k$ with $k=1$ or $2$ by Theorem 4.10. Let us prove opposition first, in both cases. If $k=1$ , two isotropic lines are not opposite if they are orthogonal. By double ergodicity of $\Gamma $ , if the opposition condition is not satisfied, then one can find a subset $B_1$ of B of full measure such that for all $b\in B_1$ and $\gamma \in \Gamma $ , $\phi (b)$ and $\phi (\gamma b)$ are orthogonal. Fix $b\in B_1$ . In particular, the span of $\{\phi (\gamma b)\}_{\gamma \in \Gamma }$ is totally isotropic (thus of dimension at most two) and $\Gamma $ -invariant, which contradicts the assumption.

Now if $k=2$ , two distinct isotropic planes are not opposite if and only if their intersection is a line. We claim that the essential image of $\phi $ is given by isotropic planes with a common line. Let $V_b=\phi (b)$ , assume the map $\phi $ is not essentially constant, and choose $V_1,V_2$ distinct isotropic planes with a common line $\ell =V_1\cap V_2$ such that almost surely $V_b\cap V_i$ is a line. If $\ell $ lies in $V_b$ almost surely, then $\ell $ is $\Gamma $ -invariant. So assume that $\ell $ is not essentially contained in $V_b$ , then there is $V_3$ in the image of $\phi $ such that $\ell $ is not in $V_3$ . So $\ell _1=V_1\cap V_3$ and $\ell _2=V_2\cap V_3$ are distinct lines and thus $V_3=\ell _1\oplus \ell _2$ lies in $V_1+V_2$ . Now, for any $b'\in B$ , if $V_{b'}$ contains $\ell $ , then $V_{b'}$ is spanned by a $\ell $ and a line in $V_3$ . If not, $V_{b'}$ meets $V_1$ and $V_2$ in two different lines. In both cases, $V_{b'}$ lies in $V_1+V_2$ . So, $V_b$ lies in $V_1+V_2$ which is thus $\Gamma $ -invariant. So we have a contradiction and thus we know that $\phi $ has the opposition property.

We conclude the proof by showing that if the image of $\phi $ lies in $\mathcal I_1$ , then there is also a $\Gamma $ -map to $\mathcal I_2$ . We rely on the beginning of the proof of Theorem 1.7 in the Zariski-dense case before the appearance of stabilizers of subspaces in some exterior power at the very end. Let us denote $\ell _b$ for the line $\phi (b)$ . In particular, we can reduce to one of the following two cases: either there is an invariant section of the field $X_b=\mathcal {X}(\ell _b^{\bot }/\ell _b,Q)$ or there is an invariant flat subfield not reduced to a point.

If there is an invariant section of the field $X_b=\mathcal {X}(\ell _b^{\bot }/\ell _b,Q)$ , then we get a map $b\mapsto V_b$ , where $V_b$ is a two-dimensional linear subspace containing $\ell _b$ and such that the signature of Q on $V_b$ is $(1,0)$ . We also know (by the same argument as in the proof of Theorem 1.7) that almost surely $V_b\cap V_{b'}$ is a positive definite line which is orthogonal to $\ell _b$ and $\ell _{b'}$ . If this intersection is essentially constant, then we have a positive definite invariant line and we are done. So assume this is not the case. We can choose $V_1,V_2$ distinct in the essential image and $V_3$ that does not contain $V_1\cap V_2$ . The same argument as in the proof of opposition shows that any $V_b$ in the essential image actually lies in $V_1+V_2$ and we have a contradiction, which shows that it cannot happen that there is an invariant section of the field $X_b=\mathcal {X}(\ell _b^{\bot }/\ell _b,Q)$ .

If there is an invariant flat subfield not reduced to a point in $X_b$ , then it is a geodesic because $X_b$ has rank one. This means, as before, that there is a map $b\mapsto V_b$ , where $V_b$ is a three-dimensional subspace of signature $(1,1)$ that contains $\ell _b$ (which is the kernel of the restriction of Q to $V_b$ ). By construction of the perspectivity $\sigma _{b,b'}$ , one has that almost surely $V_b\cap V_{b'}$ is a two-dimensional subspace of signature $(1,1)$ . If this intersection is essentially constant, then we have a two-dimensional invariant linear subspace and we are done.

If this is not the case, then as before choose $V_1,V_2$ in the essential image of the map $b\mapsto V_b$ and $V_3$ that does not contain $V_1\cap V_2$ , so $V_3\cap V_1$ and $V_3\cap V_2$ are two distinct subspaces of dimension two. In particular, their union spans $V_3$ and $V_3\leq V_1+V_2$ . Now let $V_b$ be in the essential image. For the same reason as for $V_3$ , either $V_b$ lies in $V_1+V_2$ or $V_b$ contains $V_1\cap V_2$ , but in this last case, $V_b$ meets $V_3$ in a two-dimensional subspace that contains a line not included in $V_1\cap V_2$ . So $V_b\leq (V_1\cap V_2)+V_3\leq V_1+V_2$ . Once again, we get that $V_1+V_2$ is $\Gamma $ -invariant.

If there is no invariant section nor invariant flat subfield in $(X_b)$ , then there is a map $\psi \colon b\mapsto \partial X_b$ which yields the desired map to $\mathcal I_2$ .

It is shown in [MP14, Proposition 5.5] that geometric density implies irreducibility (in the real case, but the proof works over ${\mathbf{C}}$ and $\textbf {H}$ as well). So we deduce straightforwardly the following.

Corollary 4.17. Let $\Gamma $ be a countable group with strong boundary B and let $\rho \colon {\Gamma }\to \operatorname {\mathrm{PO}}_{{\mathbf K}}(p,\infty )$ be a representation. Assume that $p\leq 2$ and $\rho $ is geometrically dense, then there is a $\Gamma $ -map $\phi \colon B\to \mathcal I_p$ such that for almost every $(b,b')\in B\times B$ , $\phi (b)$ is opposite to $\phi (b')$ .

Remark 4.18. In general, irreducibility of the representation ${\Gamma }\to \operatorname {\mathrm{PO}}_{{\mathbf K}}(p,\infty )$ implies non-elementarity and it is shown in [MP14, Proposition 5.5] that geometric density implies irreducibility. The converse of the latter implication does not hold because the embedding of $\operatorname {{\textrm O_{\mathbf{C}}}}(1,\infty )$ in $\operatorname {{\textrm O_{\mathbf {R}}}}(2,\infty )$ (given by considering the underlying real Hilbert space ${\mathcal H}_{\mathbf {R}}$ and the real part of the Hermitian form) is irreducible but not geometrically dense because a copy of ${\mathcal X}_{\mathbf{C}}(1,\infty )$ embeds equivariantly in ${\mathcal X}_{\mathbf {R}}(2,\infty )$ .

We do not expect that Theorem 4.16 holds for $p\geq 3$ but it is likely that Corollary 4.17 holds for $p\geq 3$ .

5 Bounded cohomology and the bounded Kähler class

In this section, we will recall the definitions of maximal representations, as well as adapt them to deal with infinite-dimensional symmetric spaces. Some familiarity with the basics on Kähler classes and bounded cohomology in finite dimension is advisable; the interested reader can consult, for example, [BI09].

5.1 Bounded cohomology

We recall here the notions from the theory of bounded cohomology that we will need in the paper. We refer the reader to [Mon01] for a thorough treatment.

The bounded cohomology $\text {H}_b^n(G,\textbf {R})$ of a group G is the cohomology of the complex

$$ \begin{align*}\text{C}_b^n(G,\textbf{R})^G=\bigg\{f:G^{n+1}\to \textbf{R}| f \text{ is } G\text{-invariant, }\sup_{(g_0,\ldots,g_n)\in G^{n+1}}|f(g_0,\ldots,g_n)|<\infty\bigg\}\end{align*} $$

whose coboundary operator is defined by the formula

$$ \begin{align*}df(g_0,\ldots,g_{n+1})=\sum_{i=0}^{n+1}(-1)^i f(g_0,\ldots,\hat g_i,\ldots, g_{n+1}).\end{align*} $$

The bounded cohomology of discrete groups was first introduced by Gromov [Gro82], and proved to be a useful tool in proving rigidity results, in particular because it allows detection of the properties of boundary maps. We will also exploit this feature in Proposition 5.10 below.

Despite bounded cohomology being, in general, a much wider theory then ordinary group cohomology (e.g. the third bounded cohomology of a free group is infinite-dimensional), it admits a natural homomorphism

$$ \begin{align*}c:\text{H}_b^n(G,\textbf{R})\to \text{H}^n(G,\textbf{R}),\end{align*} $$

the comparison map, induced by the inclusion of bounded cochains in ordinary cochains.

A second important advantage of bounded cohomology over ordinary group cohomology that will play a crucial role also in our work is that the $\ell ^\infty $ -norm on bounded cochain $\text {C}_b^n(G,\textbf {R})$ induces a seminorm, the Gromov norm, in bounded cohomology:

$$ \begin{align*}\|\kappa\|_{\infty}:=\inf_{[f]=\kappa}\sup_{(g_0,\ldots,g_n)\in G^n}|f(g_0,\ldots,g_n)|.\end{align*} $$

When G is a locally compact group, Burger and Monod [BM99] defined the continuous bounded cohomology $\text { H}_{cb}^n(G,\textbf {R})$ of G and showed that, in degree two, the comparison map $c:\text {H}_{cb}^2(G,\textbf {R})\to \text { H}_{c}^2(G,\textbf {R})$ is an isomorphism when G is a semisimple Lie with finite center. Here, $\text {H}_{c}^2(G,\textbf {R})$ denotes the continuous cohomology of G (a standard text about continuous cohomology is [BW00]). The result of Burger and Monod allows to give a complete description of $\text {H}_{cb}^2(G,\textbf {R})$ in case of semisimple Lie groups with finite center: the continuous cohomology $\text {H}_{c}^2(G,\textbf {R})$ can be identified with the vector space of G-invariant differential form $\Omega ^2({\mathcal X},\textbf {R})^G$ , where ${\mathcal X}$ is the symmetric space associated to G. In particular, for a simple Lie group G of non-compact type and finite center, the second continuous cohomology $\text {H}_{cb}^2(G,\textbf {R})$ is equal to $\textbf {R}\kappa ^{cb}_G$ if ${\mathcal X}$ is a Hermitian symmetric space (and $\kappa ^{cb}_G$ is then the bounded Kähler class, see below), and vanishes otherwise. In general, $\text { H}_{cb}^2(G,\textbf {R})$ is generated by the bounded Kähler classes of the Hermitian factors of ${\mathcal X}$ .

5.2 The bounded Kähler class

We now turn our attention to the bounded cohomology of the groups G of isometries of the infinite-dimensional Hermitian symmetric spaces ${\mathcal X}$ introduced in §2. Because such groups G are not locally compact, there is no well-established theory of continuous bounded cohomology; therefore, we will just work with the bounded cohomology $\text {H}_b^2(G,\textbf {R})$ . If ${\mathcal X}$ has finite rank, we can use the Kähler form to define a class in the bounded cohomology of G, precisely as in the finite-dimensional case.

Definition 5.1. The bounded Kähler class of the groups $G=\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ and $G=\operatorname {\mathrm{PO}}_{\mathbf {R}}^+(2,\infty )$ is the class $\kappa ^b_G\in \text {H}_b^2(G,\textbf {R})$ defined by the cocycle

$$ \begin{align*}C_{\omega}^x(g_0,g_1,g_2)=\frac{1}{\pi}\int_{\Delta(g_0x,g_1x,g_2x)}\omega,\end{align*} $$

where x is any base point in the corresponding symmetric space ${\mathcal X}$ , $\Delta (g_0x,g_1x,g_2x)$ is the geodesic triangle with vertices $(g_0x,g_1x,g_2x)$ , and $\omega $ is the Kähler form normalized such that the minimum of the holomorphic sectional curvature is $-1$ .

The fact that $\kappa ^b_G$ is independent on x is proved below.

Remark 5.2. Let $i:H=\operatorname {\mathrm{SU}}(p,q)\to G=\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ be a standard embedding. As before, we denote by $\kappa ^{cb}_H\in \text { H}_{cb}^2(H,\textbf {R})$ the generator corresponding, under the natural isomorphism, to the bounded Kähler class, and we denote by $\kappa ^{b}_H\in \text {H}_b^2(H,\textbf {R})$ the image of $\kappa ^{cb}_H$ under the map induced by the inclusion of continuous bounded cochains in bounded cochains. It follows from the definition that $i^*{\kappa ^b_G}=\kappa ^b_H$ .

Lemma 5.3. The class $\kappa ^b_G$ is well defined. Furthermore,

$$ \begin{align*}\|\kappa^b_G\|_{\infty}=\mathrm{rk}({\mathcal X}),\end{align*} $$

where ${\mathcal X}$ is the symmetric space associated to G. In particular, $\kappa ^b_G$ is not zero.

Proof. The cocycle $C_{\omega }^x$ has norm bounded by $\text {rk}({\mathcal X})$ because the three points $(g_0x,g_1x,g_2x)$ lie on some isometrically embedded totally geodesic copy of ${\mathcal X}_{\mathbf{C}}(p,2p)$ (respectively ${\mathcal X}_{\mathbf {R}}(2,4)$ ) and therefore the sharp bound of the integral computed in [DT87] applies. Furthermore, the class $\kappa ^b_G$ does not depend on the choice of the base point x because for any other point y, the difference $C_{\omega }^x-C_{\omega }^y$ is the coboundary of the function

$$ \begin{align*}f_{\omega}^{x,y}(g_0,g_1)=\frac{1}{\pi}\int_{\Delta(g_0x,g_1x,g_1y)}\omega+\frac{1}{\pi}\int_{\Delta(g_0x,g_1y,g_0y)}\omega,\end{align*} $$

which, again, is bounded because the four points $(g_0x,g_1x,g_0y,g_1y)$ lie on some isometrically embedded totally geodesic copy of ${\mathcal X}_{\mathbf{C}}(p,3p)$ (respectively ${\mathcal X}_{\mathbf {R}}(2,6)$ ). Observe that for any triple $(g_0,g_1,g_2)$ , the value $C_{\omega }^y(g_0,g_1,g_2)-C_{\omega }^x(g_0,g_1,g_2)+df_{\omega }^{x,y}(g_0,g_1,g_2)$ is the integral of the closed form $\omega $ on a triangulation of the triangular prism with the bottom face $\Delta (g_0x,g_1x,g_2x)$ and upper face $ \Delta (g_0y,g_1y,g_2y).$ This is a closed polyhedral surface contained in a finite-dimensional subspace; therefore, the integral of $\omega $ over it vanishes.

To conclude the proof, we therefore only need to show that $\|\kappa ^b_G\|_{\infty }\geq \text {rk}({\mathcal X})$ . For this purpose, let $\Gamma $ denote the fundamental group of a surface and let us consider the homomorphism $i:{\Gamma }\to \operatorname {\mathrm{SU}}(p,p)\to \operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ (respectively $i:{\Gamma }\to \operatorname {\textrm {SO}}^+(2,2)\to \operatorname {\mathrm{PO}}_{\mathbf {R}}^+(2,\infty )$ ), in which the inclusion ${\Gamma }\leq \operatorname {\mathrm{SU}}(1,1)\to \operatorname {\mathrm{SU}}(p,p)$ (respectively ${\Gamma }\to \operatorname {\textrm {SO}}(2,2)$ ) is such that the diagonal inclusion of the Poincaré disk in a maximal polydisk is equivariant. It follows from [BILW05, Example 3.9] together with Remark 5.2 that $\|i^*\kappa ^b_G\|_{\infty }=\text {rk}({\mathcal X})$ . Because the pullback in bounded cohomology is clearly norm non-increasing, the result follows.

5.3 The Bergmann cocycle

In the study of rigidity properties of maximal representations, it will be useful to have a different representative of the bounded Kähler class. Such a representative will depend only on the action on a suitable boundary of the symmetric space. We distinguish two cases.

When dealing with the groups $\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ , the new representative will depend on the choice of a point $V\in \mathcal I_p$ the set of maximal isotropic subspaces of ${\mathcal H}(p,\infty )$ . Recall that every triple $(V_0,V_1,V_2)\in (\mathcal I_p)^3$ is contained in a finite-dimensional subspace (of dimension at most $3p$ ). This implies that the Bergmann cocycle studied in [Cle02, Cle07]. In [Cle02, Cle07], this cocycle is referred to as the generalized Maslov index. We chose to denote this cocycle as a Bergmann cocycle, following [BIW09, §3.2] for $\operatorname {\mathrm{SU}}(p,3p)$ extends to a strict $\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ -invariant cocycle

$$ \begin{align*}\beta_{\mathbf{C}}:\mathcal I_p^3\to [-\text{rk}({\mathcal X}),\text{rk}({\mathcal X})]\end{align*} $$

with the property that if $|\beta _{{\mathbf{C}}}(V_0,V_1,V_2)|=\text {rk}({\mathcal X})$ , then $V_0,V_1,V_2$ are contained in a $2p$ -dimensional subspace of signature $(p,p)$ and are pairwise transverse.

While we will not recall the explicit definition of the Bergmann cocycle (we refer to the aforementioned papers), we record its most important property.

Lemma 5.4. For every $V\in \mathcal I_p$ , the cocycle $C_{\beta }^V$ defined by

$$ \begin{align*}C_{\beta}^V(g_0,g_1,g_2)=\beta_{{\mathrm{\mathbf{C}}}}(g_0V,g_1V,g_2V)\end{align*} $$

represents the bounded Kähler class.

Proof. Because any 4-tuple $(V_0,V_1,V_2,V_3)\in \mathcal I_p^4$ is contained in a finite-dimensional subspace of ${\mathcal H}(p,\infty )$ , it follows from [Cle07, Theorem 5.3] that the cocycle $C_{\beta }^V$ is a strict alternating bounded cocycle, cohomologous to $C_{\omega }^x$ : the difference of the cocycles is the coboundary of a function defined similarly to the function $f_{x,y}$ in the proof of Lemma 5.3, but integrating on simplices with some ideal vertices.

Remark 5.5. It is worth remarking that if G is a (finite dimensional) Hermitian Lie group, the cocycles $C_{\omega }^x$ and $C_{\beta }^V$ also define a class $\kappa ^{cb}_G$ in the continuous bounded cohomology $\text {H}_{cb}^2(G,\textbf {R})$ . This class generates the continuous bounded cohomology $\text {H}_{cb}^2(G,\textbf {R})$ for simple groups of Hermitian type.

In the case of the group $\operatorname {\mathrm{PO}}_{\mathbf {R}}^+(2,\infty )$ , the same construction works except that the boundary of ${\mathcal X}_{\mathbf {R}}(2,n)$ on which the Bergmann cocycle is defined is $\cal {I}_1(2,n)$ and not $\cal {I}_2(2,n)$ . Thus, the Bergmann cocycle for $\operatorname {\mathrm{PO}}_{\mathbf {R}}^+(2,\infty )$ is a map $\beta _{\mathbf {R}}:\mathcal I_1(2,\infty )^3\to \{-2,0,2\}$ . The fact that, in this case, the Bergmann cocycle only assumes a discrete set of possible values reflects the fact that $\operatorname {{\textrm O_{\mathbf {R}}}}(2,p)$ is of tube type. It is worth remarking that, in this case, the Bergmann cocycle is only preserved by the connected component of the identity in $\operatorname {{\textrm O_{\mathbf {R}}}}(2,\infty )$ , denoted by $\operatorname {{\textrm O_{\mathbf {R}}}}^+(2,\infty )$ .

It is possible to give an explicit description (based upon [Cle04, §6]) of the value of the Bergmann cocycle for triples of pairwise opposite points in $\mathcal I_1(2,\infty )$ . For this, we need to choose representatives $\overline x,\overline y,\overline z$ of the classes $x,y,z$ such that $Q(\overline x,\overline z)<0$ and $Q(\overline x,\overline y)<0$ ; furthermore, given an isotropic vector $\overline x$ in ${\mathcal H}$ , we denote by $[\overline x]$ the vector in $\textbf {R}^2=\langle e_1,e_2\rangle $ , which corresponds to the orthogonal projection (with respect to Q) of $[\overline x]$ , and we endow $\textbf {R}^2=\langle e_1,e_2\rangle $ with its canonical orientation, which allows us to determine if a triple of pairwise distinct non-zero vectors is positively or negatively oriented. We then define (here $\textrm{or}$ denotes the orientation):

$$ \begin{align*}\left\{\begin{array}{@{}ll} \beta_{\mathbf{R}}(x,y,z)=0 \quad&\text{if } Q|_{\langle x,y,z\rangle} \text{ has signature } (1,2),\\ \beta_{\mathbf{R}}(x,y,z)=2 \quad&\text{if } Q|_{\langle x,y,z\rangle} \text{ has signature} (2,1), \text{ and } \textbf{or} ([\overline x],[\overline y],[\overline z])=+,\\ \beta_{\mathbf{R}}(x,y,z)=-2 \quad&\text{if } Q|_{\langle x,y,z\rangle} \text{ has signature } (2,1), \text{ and } \textbf{or} ([\overline x],[\overline y],[\overline z])=-.\\ \end{array}\right.\end{align*} $$

One checks that the value of $\beta _{\mathbf {R}}$ does not depend on the choices involved and $\beta _{\mathbf {R}}$ coincides with the Bergmann cocycle.

To unify the notation, we will denote, from now on, by $\mathcal S_G$ the spaces $\mathcal S_{\operatorname {{\textrm O_{\mathbf {R}}}}(2,\infty )}:=\mathcal I_1(2,\infty )$ and $\mathcal S_{\operatorname {{\textrm O_{\mathbf{C}}}}(p,\infty )}:=\mathcal I_p(p,\infty )$ . Similarly, when this will not seem to generate confusion, we will simply use the letter $\beta $ for the cocycles that we denoted before as $\beta _{\mathbf {R}}$ (respectively $\beta _{\mathbf{C}}$ ).

5.4 Maximal representations

Let $\Gamma \leq \operatorname {\mathrm{SU}}(1,n)$ be a lattice. We denote by

$$ \begin{align*}T_b^*:\text{H}_b^2(\Gamma,\textbf{R})\to \text{H}_{cb}^2(\operatorname{\textrm{SU}}(1,n),\textbf{R})\end{align*} $$

the transfer map, as defined in [BI09, §2.7.2]: this is a left inverse of the restriction map $i^*:\text { H}_{cb}^2(\operatorname {\mathrm{SU}}(1,n),\textbf {R})\to \text {H}_b^2({\Gamma },\textbf {R})$ that has norm one. Recall that $ \text {H}_{cb}^2(\operatorname {\mathrm{SU}}(1,n),\textbf {R})\cong \textbf {R}$ and is generated by the bounded Kähler class of the group $\operatorname {\mathrm{SU}}(1,n)$ [BM99, Lemma 6.1]. In this section, we will denote the bounded Kähler class of the group $\operatorname {\mathrm{SU}}(1,n)$ by $\kappa ^{cb}_n$ , to avoid confusion with the other Kähler classes, and simplify the notation.

Definition 5.6. Let $G\in \{\operatorname {\mathrm{PO}}_{\mathbf {R}}(2,\infty ),\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )\}$ and let $\rho :{\Gamma }\to G$ be a homomorphism. The Toledo invariant of the representation $\rho $ is the number $i_\rho $ such that

(1) $$ \begin{align}T_b^*\rho^*\kappa^b_G =i_\rho \kappa_n^{cb}.\end{align} $$

Observe that the absolute value $|i_\rho |$ of the Toledo number is bounded by $\text {rk}(G)$ because both the transfer map and the pullback are norm non-increasing. This inequality is often referred to as the generalized Milnor–Wood inequality. In analogy with [BI09], we say the following.

Definition 5.7. The representation $\rho $ is maximal if $|i_\rho |=p$ .

As in the finite-dimensional case, it follows from the definition that the restriction of a maximal representation to a finite index subgroup is also maximal.

Lemma 5.8. The restriction of a maximal representation $\rho :{\Gamma }\to G$ to a finite index subgroup $\Lambda <\Gamma $ is maximal.

Proof. Indeed denoting by $T_{b,\Lambda }^*$ (respectively $T_{b,\Gamma }^*$ ) the transfer map and by $\iota ^*:\text {H}_b^2(\Gamma ,\textbf {R})\to \text { H}_b^2(\Lambda ,\textbf {R})$ the isometric injection induced in bounded cohomology by the inclusion $\iota :\Lambda \to \Gamma $ [Mon01, Proposition 8.6.2], one gets $T_{b,\Gamma }^*=T_{b,\Lambda }^*\iota ^*$ .

Also the following fact descends directly from the definition, but is very useful in understanding geometric properties of maximal representations.

Proposition 5.9. Let $\rho :\Gamma \to \operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ be a maximal representation. Then there is no fixed point at infinity for $\rho $ .

Proof. Assume by contradiction that $\rho (\Gamma )$ fixes an isotropic subspace V, and choose any maximal isotropic subspace $V'$ containing V. The cocycle $C_{\beta }^V\circ \rho $ represents the class $\rho ^*\kappa _G^b$ and, because maximal triples consist of pairwise transverse subspaces, has norm strictly smaller than p, thus leading to a contradiction.

We conclude this subsection observing that, as in the finite-dimensional case, the pullback in bounded cohomology can be realized through boundary maps.

Proposition 5.10. Let $H=\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ and $\rho :\Gamma \to H$ be a maximal representation. If there exists a measurable $\rho $ -equivariant boundary map $\phi :\mathcal I_1(1,n)\to \mathcal S_H$ , then for every triple of pairwise distinct points $(x,y,z)\in \mathcal I_1(1,n)$ , it holds that

$$ \begin{align*}\mathrm{rk}(H) \beta_{\mathcal I_1(1,n)}(x,y,z)=\int_{\Gamma \backslash \operatorname{\textrm{SU}}(1,n)}\beta(\phi(gx),\phi(gy),\phi(gz))\,{d}\mu(g),\end{align*} $$

where $\mu $ on $\operatorname {\mathrm{SU}}(1,n)/\Gamma $ is the unique $\operatorname {\mathrm{SU}}(1,n)$ -invariant probability measure.

Proof. We use the formula in [BI09, Proposition 2.38]. Let $G=\operatorname {\mathrm{SU}}(1,n)$ , $L=G'=\Gamma $ , $X=\mathcal S_H$ with its Borel $\sigma $ -algebra. Let $\kappa '=\rho ^*\kappa ^b_{G}\in \text {H}_b^2(\Gamma ,\textbf {R})$ be the pullback of the bounded Kähler class and $\kappa = T_b^*\rho ^*\kappa ^b_{H} \in \text {H}_{cb}^2(\operatorname {\mathrm{SU}}(1,n),\textbf {R})$ . Because $\text {rk}(H)\beta _{\mathcal I_1(1,n)}$ and $\beta $ are strict alternating bounded cocycles representing respectively $\kappa $ and $\kappa '$ , equation (2.12) in [BI09, Proposition 2.38] yields that

$$ \begin{align*}(x,y,z)\mapsto \text{rk}(H)\beta_{\mathcal I_1(1,n)}(x,y,z)-\int_{\Gamma\backslash \operatorname{\textrm{SU}}(1,n)}\beta(\phi(gx),\phi(gy),\phi(gz))\,{d}\mu(g)\end{align*} $$

is a coboundary in $L^\infty (X^3)$ . By [BI09, Remark 3.1], the coboundary actually vanishes and thus the equality claimed holds almost surely. Now, because both terms of the equation are everywhere defined, are G-invariant, and satisfy the cocyle relation, the same argument as in [Poz15, Lemma 2.11] proves that the equality holds for every triple of pairwise distinct points $(x,y,z)\in \mathcal I_1(1,n)$ .

5.5 Tight homomorphisms and tight embeddings

Burger, Iozzi, and Wienhard introduced, in [BIW09], the notion of tight homomorphism between Hermitian Lie groups and analogously tight embeddings between Hermitian symmetric spaces: this is of fundamental importance in the study of maximal representations because, on the one hand, tight homomorphisms between Lie groups can be completely classified, and on the other, the inclusion of the Zariski closure of the image of a maximal representation is tight; this allows, in the finite-dimensional setting, to reduce the study of maximal representations to Zariski-dense maximal representations, for which construction of boundary maps is much easier.

In analogy with [BIW09, Definition 2.4], we define the following.

Definition 5.11. Let ${\mathcal X}$ and $\mathcal {Y}$ be (possibly infinite-dimensional) Hermitian symmetric spaces of non-compact type with Kähler forms $\omega _{\mathcal X}$ and $\omega _{\mathcal {Y}}$ associated to the Riemannian metrics of minimal holomorphic sectional curvature $-1$ . A totally geodesic embedding $f\colon \mathcal {Y}\to {\mathcal X}$ is tight if

$$ \begin{align*}\sup_{\Delta\subset \mathcal{Y}}\int_\Delta f^*\omega_{\mathcal X}=\sup_{\Delta\subset \mathcal{X}}\int_\Delta \omega_{\mathcal X}.\end{align*} $$

Let H be any group and G be the isometry group of a (possibly infinite-dimensional) Hermitian symmetric space ${\mathcal X}_G$ . Let us endow G with the topology of pointwise convergence, that is, the coarsest topology on G such that $g\mapsto gx$ is continuous for any $x\in {\mathcal X}_G$ . Because ${\mathcal X}_G$ is a complete separable metric space, it is well known that G is Polish for this topology [Kec95, §9.B]. If $\iota :H\to G$ is a continuous homomorphim (that is, the action of H on ${\mathcal X}_G$ is continuous), then we denote by $\iota ^*(\kappa ^b_G)$ the continuous bounded cohomology class of the pullback of the Kähler cocycle. Let us observe that this cocycle is continuous because the integration depends continuously on the vertices of the triangle. We say that $\iota $ is tight if

$$ \begin{align*}\|\iota^*(\kappa^b_G)\|_{\infty}=\|\kappa^b_G\|_{\infty}.\end{align*} $$

Assume that, in Definition 5.11, H is the connected component of the isometry group of an irreducible Hermitian symmetric space of finite dimension and consider the homomorphism $\iota \colon H\to G$ . Because geodesic triangles are contained in finite-dimensional symmetric spaces, the tightness of $\iota $ is equivalent to the requirement that the inclusion ${\mathcal X}_H\to {\mathcal X}_G$ of the symmetric spaces associated to G and H is tight [BIW09, Corollary 2.16].

Lemma 5.12. The inclusion ${\mathcal X}_H\to {\mathcal X}_G$ of a finite-dimensional totally geodesic symmetric subspace is tight if and only if the inclusion $\iota :H\to G$ is tight.

Remark 5.13. If the inclusion ${\mathcal X}_H\to {\mathcal X}_G$ is totally geodesic, isometric, and holomorphic, then the pullback, through the equivariant group homomorphism $\iota \colon H\to G$ of the bounded Kähler class, is clearly the bounded Kähler class. This provides many examples of tight maps: whenever the symmetric spaces have the same rank, the homomorphism is tight.

Let $\rho :\Gamma \to G$ be a representation of a lattice in $\operatorname {\mathrm{SU}}(1,n)$ and assume that the symmetric space ${\mathcal X}_G$ associated to G has rank $p\in \mathbf{N}$ . Because both pullback and transfer maps are norm non-increasing, and $\|\kappa _n^{cb}\|=1$ , we deduce that $|i_\rho |\leq \|\rho ^*\kappa ^b_G\|_{\infty }$ , where $i_\rho $ is defined by equation (1). In particular, if the representation $\rho $ is maximal, then $\|\rho ^*\kappa ^b_G\|_{\infty }=p$ . The same argument gives the following.

Lemma 5.14. Assume that a maximal representation $\rho :{\Gamma }\to G$ preserves a totally geodesic Hermitian symmetric subspace $\mathcal Y\subset {\mathcal X}_G$ . Then the inclusion $\mathcal Y\to {\mathcal X}_G$ is tight.

5.6 Reduction to geometrically dense maximal representations

In this subsection, we explain how to reduce the understanding of maximal representations to geometrically dense maximal representations.

Recall that the totally geodesic subspaces of ${\mathcal X}_{\mathbf{C}}(p,\infty )$ are products of irreducible factors that are either finite-dimensional or are isomorphic to either ${\mathcal X}_{\mathbf{C}}(q,\infty )$ or ${\mathcal X}_{\mathbf {R}}(q,\infty )$ or ${\mathcal X}_{\mathbf {H}}(q,\infty )$ [Duc15a, Corollary 1.9].

Proposition 5.15. Let $\rho :\Gamma \to \operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ be a maximal representation. There is a minimal $\Gamma $ -invariant totally geodesic subspace $\mathcal Y$ of ${\mathcal X}_{\mathbf{C}}(p,\infty )$ . This space $\mathcal Y$ splits isometrically as a direct product $\mathcal Y=\mathcal Y_1\times \cdots \times \mathcal Y_k$ (possibly reduced to a unique factor), and for each i, either $\mathcal Y_i$ is finite-dimensional Hermitian, or it is isometric to ${\mathcal X}_{\mathbf{C}}(m,\infty )$ and the restricted representation $\rho _i:{\Gamma }\to \operatorname {\mathrm{Isom}}(\mathcal Y_i)$ is maximal and geometrically dense.

Proof. Because the representation is maximal, there is no fixed point at infinity (Proposition 5.9). Thus there is a minimal totally geodesic $\Gamma $ -invariant subspace $\mathcal Y$ (otherwise [Duc13, Proposition 4.4] would yield a fixed point at infinity). Because $\mathcal Y$ is a totally geodesic subspace of ${\mathcal X}_{\mathbf{C}}(p,\infty )$ , it is a symmetric space of non-positive curvature operator and finite rank. Thus, $\mathcal Y$ decomposes as a product $\mathcal Y=\mathcal Y_0\times \mathcal Y_1\times \cdots \times \mathcal Y_k$ , where each $\mathcal Y_k$ is a symmetric space of finite dimension of non-compact type, the Euclidean de Rham factor, or the symmetric space associated to some $\operatorname {{\textrm O_{\mathbf {K}}}}(l,\infty )$ with ${\textbf K}=\textbf {R}$ , ${\mathbf{C}}$ or $\textbf {H}$ [Duc15a, Corollary 1.10]. Up to passing to a finite index subgroup, we may assume that $\Gamma $ preserves each factor of this splitting (see Lemma 5.8). Because $\mathcal Y$ is minimal as $\Gamma $ -invariant totally geodesic subspace, the induced action $\Gamma \curvearrowright \mathcal Y_i$ is minimal as well.

Recall that a geodesic segment in $\mathcal Y$ has the form $\sigma (t)=(\sigma _1(t),\ldots ,\sigma _k(t))$ , where each $\sigma _i$ is a curve with constant speed (which may vary from factor to factor). Because the inclusion $\mathcal Y\subset {\mathcal X}$ is tight (Lemma 5.14), we have

$$ \begin{align*}\sup_{\Delta\subset \mathcal Y}\int_\Delta \omega_{\mathcal X}=\sup_{\Delta\subset {\mathcal X}}\int_\Delta \omega_{\mathcal X},\end{align*} $$

where $\Delta $ is a geodesic triangle. Let $\Delta _i$ be the projection of $\Delta $ to the factor $\mathcal Y_i$ . The triangle $\Delta _i$ is completely determined by three points. If $\mathcal Y_i$ has infinite dimension, these three points are given by three positive definite linear subspaces and thus are included in some standard embedding of ${\mathcal X}_{{\mathbf K}}(l,2l)$ in $\mathcal Y_i$ . We denote by $\mathcal Y_i^0$ any $\mathcal Y_i$ , if the subspace already has finite dimension or the image of a standard embedding of ${\mathcal X}_{{\mathbf K}}(l,2l)$ in $\mathcal Y_i$ . Finally, we denote by $\mathcal Y^0$ the product $\mathcal Y_0^0\times \cdots \times \mathcal Y_k^0$ . The symmetric space $\mathcal Y^0$ has finite dimension, and, because the isometry group of ${\mathcal X}_{{\mathbf K}}(p,\infty )$ acts transitively on standard embeddings of ${\mathcal X}_{\mathbf{C}}(l,2l)$ , we have

$$ \begin{align*} \sup_{\Delta\subset \mathcal Y^0}\int_\Delta \omega=\sup_{\Delta\subset \mathcal Y}\int_\Delta \omega=\sup_{\Delta\subset {\mathcal X}}\int_\Delta \omega. \end{align*} $$

Now $\mathcal Y^0$ lies in some standard copy ${\mathcal X}^0$ of ${\mathcal X}_{\mathbf{C}}(p,N)$ for $N\geq p$ (Lemma 3.11) and thus the embedding of $\mathcal Y^0$ in ${\mathcal X}^0$ is tight and we can apply [BIW09, Theorem 7.1]. So, we know that each $\mathcal Y_i^0$ is Hermitian of non-compact type and the Euclidean de Rham factor is trivial. In particular, all $\mathcal Y_i$ of finite dimension are Hermitian and the infinite-dimensional ones are a priori ${\mathcal X}_{\mathbf{C}}(m,\infty )$ and ${\mathcal X}_{\mathbf {R}}(2,\infty )$ but the latter is impossible. To see that, assume by contradiction that there is factor ${\mathcal X}_{\mathbf {R}}(2,n)$ in $\mathcal Y^0$ . Then the classification of tight embeddings obtained in [HP14, §5.2] implies that the image of the tube-type factors through the embedding lies in some totally geodesic copy of ${\mathcal X}_{\mathbf{C}}(m,m)$ . By considering the rank, we have $m\leq p$ and by considering the dimension, this gives a finite bound on n. Thus there is no factor ${\mathcal X}_{\mathbf {R}}(2,\infty )$ in $\mathcal Y$ .

Finally, the fact that each representation $\rho _i$ is maximal is standard (see for example [BIW09, Lemma 2.6(4)]).

6 Representations in $\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$

We first focus on the case where the target is $\operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ . In this case, we prove that the representation preserves a finite-dimensional totally geodesic subspace. Thus, in particular, if the domain ${\Gamma }$ is a lattice in $\operatorname {\mathrm{SU}}(1,n)$ for $n\geq 2$ , super-rigidity of maximal representations $\rho :{\Gamma }\to G$ , where G is a Hermitian Lie group applies [BI08, KM17, Poz15]. To this aim we need a good understanding of the geometry of the boundaries $\mathcal I_1(1,n)$ and $\mathcal I_p(p,\infty )$ . It can be noted that the only case that is still open is the case of representations of non-uniform lattices in groups of tube type.

6.1 The geometry of the boundary of ${{\mathcal X}_{\mathbf{C}}(1,n)}$

Recall that a chain $\mathcal C\subseteq \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ is the boundary of a totally geodesic holomorphic disk $\mathcal D\subseteq {{\mathcal X}_{\mathbf{C}}(1,n)}$ , and is the intersection of $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}\subseteq {\mathbf{C}}\textbf {P}^n$ with a complex projective line in ${\mathbf{C}}\textbf {P}^n$ . For this reason, a chain is uniquely determined by two points belonging to it. Given two distinct points $x,y\in \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ , we will denote by $\mathcal C_{x,y}$ the unique chain containing the points x and y. More generally, for every k-dimensional subspace $\textbf {P} V\subseteq {\mathbf{C}}\textbf {P}^n$ that intersects ${{\mathcal X}_{\mathbf{C}}(1,n)}$ , the subspace $\textbf {P} V$ intersects ${{\mathcal X}_{\mathbf{C}}(1,n)}$ in a totally geodesic submanifold isometric to ${\mathcal X}_{\mathbf{C}}(1,k)$ and intersects $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ in a $(2k-1)$ -dimensional sphere $\partial {\mathcal X}_{\mathbf{C}}(1,k)$ . Following [Gol99], we will call any such sphere a k-hyperchain.

Part of the work of [Poz15] was aimed at showing that a similar picture exists in higher rank: any two transverse subspaces $X,Y\in \mathcal I_p(p,\infty )$ determine a $2p$ -dimensional subspace $\langle X,Y\rangle $ and therefore a finite-dimensional totally geodesic subspace ${\mathcal X}_{\mathbf{C}}(p,p)\subset {\mathcal X}_{\mathbf{C}}(p,\infty )$ as well as a subset $\mathcal I_p(p,p)\subset \mathcal I_p(p,\infty )$ . As in [Poz15], we will refer to these subsets as p-chains or merely chains.

Here and in the rest of the article, when we will deal with differentiable manifolds, almost surely will mean for a set of full measure in the Lebesgue measure class. We say that a measurable map $\phi :\partial {{\mathcal X}_{\mathbf{C}}(1,n)}\to \mathcal I_p(p,\infty )$ almost surely maps chains to chains if for almost every chain $\mathcal C\subseteq \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ , there is a p-chain $\mathcal T\subset \mathcal I_p(p,\infty )$ such that for almost every point $x\in \mathcal C$ , $\phi (x)\in \mathcal T$ . In this case, we say that the chain $\mathcal C$ is generic for $\phi $ . To guarantee that a measurable map $\phi $ almost surely maps chains to chains, it is enough to check that for almost every pair $(x,y)\in \partial {{\mathcal X}_{\mathbf{C}}(1,n)}\times \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ , the subspaces $\phi (x)$ and $\phi (y)$ are transverse and that for almost every $z\in \mathcal C_{x,y}$ , the subspace $\phi (z)$ is contained in $\langle \phi (x),\phi (y)\rangle $ (this statement can be found e.g. in [Poz15, Lemma 4.2]). In this case, we say that the pair $(x,y)$ is generic for $\phi $ .

A consequence of Proposition 5.10 is the following.

Corollary 6.1. Let ${\Gamma }<\operatorname {\mathrm{SU}}(1,n)$ be a lattice. Assume that a representation $\rho :{\Gamma }\to \operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ is maximal and admits an equivariant boundary map $\phi :\partial {\mathcal X}_{\mathbf{C}}(1,n)\to \mathcal I_p(p,\infty )$ . Then the boundary map $\phi $ almost surely maps chains to chains.

Proof. Observe that, because $\operatorname {\mathrm{SU}}(1,1)$ acts transitively on positively oriented triples in $\mathcal I_1(1,1)$ , there are precisely two $\operatorname {\mathrm{SU}}(1,n)$ orbits of pairwise distinct triples of points on a chain. Because the equality in Proposition 5.10 holds for every triple $(x,y,z)$ , we deduce that for almost every triple $(x,y,z)$ on a chain, the triple $(\phi (x),\phi (y),\phi (z))$ is contained in a p-chain and consists of transverse points. Hence, $\phi $ almost surely maps chains to chains.

The purpose of the rest of the section will then be to show the following proposition.

Proposition 6.2. If $\phi :\partial {{\mathcal X}_{\mathbf{C}}(1,n)}\to \mathcal I_p(p,\infty )$ is measurable and almost surely maps chains to chains, then there exists a finite-dimensional, totally geodesic subspace ${\mathcal X}_{p,np}\subset {\mathcal X}_{p,\infty }$ such that $\phi (\partial {{\mathcal X}_{\mathbf{C}}(1,n)})\subset \partial {\mathcal X}_{p,np}$ up to discarding a null subset of $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ .

The proof of Proposition 6.2 is a measurable version of an easy geometric construction (Lemma 6.6). Compare with [BI08, Poz15] for similar statements and arguments. To prove the proposition, we will need several easy lemmas, the first of which is a straightforward consequence of Fubini’s theorem.

Lemma 6.3. Let $A,B$ be differentiable manifolds and $\pi :A\to B$ be a smooth fibration. Then:

1. (1) if $\mathcal O\subseteq B$ has full measure, then $\pi ^{-1}(\mathcal O)\subset A$ has full measure;

2. (2) if $\mathcal Y\subseteq A$ has full measure, then for almost every $x\in B$ , $\mathcal Y\cap \pi ^{-1}(x)$ has full measure in $\pi ^{-1}(x)$ .

In the proof of Proposition 6.2, we will argue by induction on the dimension n of $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ . In particular, to have at our disposal the inductive step, we will need the following.

Lemma 6.4. If the measurable map $\phi :\partial {{\mathcal X}_{\mathbf{C}}(1,n)}\to \mathcal I_p(p,\infty )$ almost surely maps chains to p-chains, then for every $1\leq k\leq n$ and for almost every k-hyperchain $\partial {\mathcal X}_{\mathbf{C}}(1,k)\subset \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ , the restriction $\phi |_{\partial {\mathcal X}_{\mathbf{C}}(1,k)}:\partial {\mathcal X}_{\mathbf{C}}(1,k)\to \mathcal I_p(p,\infty )$ almost surely maps chains to chains.

Proof. This is an application of Lemma 6.3: consider the configuration spaces

$$ \begin{align*}\begin{array}{@{}l} \mathcal E_1^k=\{(C,X)|\; C\text{ is a chain}, X\text{ is a } k\text{-hyperchain}, C\subseteq X\},\\ \mathcal E_0=\{C|\;C\text{ is a chain}\}.\end{array}\end{align*} $$

Clearly, for every k, there is a smooth surjection $\mathcal E_1^k\to \mathcal E_0$ . In particular, the subset $\mathcal Y\subset \mathcal E_1^k$ consisting of pairs $(C,X)$ such that C is generic for $\phi $ has full measure. Lemma 6.3(2) implies the desired statement.

In the inductive step, we will need to increase the dimension of $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ by one. For this purpose, the following additional configuration spaces will be handy:

$$ \begin{align*}\begin{array}{l} \mathcal F_0=\{(C,X)|\; C \text{ is a chain, } X\text{ is a } (n-1)\text{-hyperchain, } C\cap X\; \text{is a point}\};\\ \mathcal F_1=\{(C,X,c,x)|\; (C,X)\in\mathcal F_0,\; c\in C,\; x\in X \};\\ \mathcal F_2=\{(C,X,c)|\;(C,X)\in\mathcal F_0, c\in C\};\\ \mathcal F_3=\{(C,X,x)|\;(C,X)\in\mathcal F_0, x\in X\}. \end{array}\end{align*} $$

Lemma 6.5. Assume $\phi :\partial {{\mathcal X}_{\mathbf{C}}(1,n)}\to \mathcal I_p(p,\infty )$ almost surely maps chains to chains. Then for almost every pair $(C,X)\in \mathcal F_0$ , the following hold:

1. (1) the chain C is generic for $\phi $ ;

2. (2) for almost every pair $(c,x)\in C\times X$ , the pair $(c,x)$ is generic for $\phi $ ;

3. (3) for almost every point $c\in C$ , the pair $(c, C\cap X)$ is generic for $\phi $ ;

4. (4) for almost every point $x\in X$ , the pair $(x, C\cap X)$ is generic for $\phi $ .

Proof. Because the intersection of finitely many full measure subsets has full measure, it is enough to verify that each condition holds on a full measure set. The pairs for which condition (1) holds have clearly full measure: by assumption, almost every chain is generic and $\mathcal F_0$ smoothly fibers over the set of all chains.

To verify condition (2) observe that, becuase $\phi $ almost surely maps chains to chains, the set of pairs $(c,x)\in \partial {{\mathcal X}_{\mathbf{C}}(1,n)}\times \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ that are generic for $\phi $ has full measure. Consider now the forgetful map $\pi :\mathcal F_1\to \partial {{\mathcal X}_{\mathbf{C}}(1,n)}\times \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ . If we restrict to the open dense subset of $\mathcal F_1$ consisting of 4-tuples $(C,X,c,x)$ , such that $c, x$ and $C\cap X$ are pairwise distinct, $\pi $ gives a surjective fibration onto the (open and dense) set of transverse pairs in $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}\times \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ . In particular, we deduce from Lemma 6.3(1) that the set of 4-tuples $(C,X,c,x)\in \mathcal F_1$ , such that $(c,x)$ is generic for $\phi $ , has full measure in $\mathcal F_1$ . Because $\mathcal F_1\to \mathcal F_0$ is a smooth fibration, the statement is then a direct consequence of Lemma 6.3(2).

To verify that the last two conditions hold on a full measure set as well, we use a similar argument for the fibrations $\mathcal F_2\to \partial {{\mathcal X}_{\mathbf{C}}(1,n)}\times \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ and $\mathcal F_3\to \partial {{\mathcal X}_{\mathbf{C}}(1,n)}\times \partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ given respectively by $(C,X,c)\mapsto (C\cap X,c)$ and $(C,X,x)\mapsto (C\cap X,x)$ .

The inductive step will be based on the following construction.

Lemma 6.6. For any pair $(C,X)$ in $\mathcal F_0$ , the union

$$ \begin{align*}S=\bigcup_{\substack{c\in C\\x\in X}}\mathcal C_{c,x}\end{align*} $$

contains an open and dense subset of $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ .

Proof. We work in the Heisenberg model for $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ in which the intersection point $C\cap X$ corresponds to $\infty $ . It is well known that in this model, isomorphic to ${\mathbf{C}}^{n-1}\ltimes \textbf {R}$ , a chain W corresponds to either a vertical line or to a topological circle that projects to an Euclidean circle E contained in an affine complex line $L\subset {\mathbf{C}}^{n-1}$ . Moreover, denoting by $\pi :{\mathbf{C}}^{n-1}\ltimes \textbf {R}\to {\mathbf{C}}^{n-1}$ the projection, for every Euclidean circle $E\subset {\mathbf{C}}^{n-1}$ , and every point $x\in \pi ^{-1}(E)$ , there exists a unique chain W containing x and satisfying $\pi (W)=E$ [Gol99, §4.3].

Because we chose the Heisenberg model in which $C\cap X$ corresponds to $\infty $ , (see Figure 1) the chain C corresponds to a vertical line (preimage of the point $p_C\in {\mathbf{C}}^{n-1}$ ), and the $(n-1)$ -hyperchain X corresponds to the preimage under $\pi $ of a $(n-2)$ -dimensional affine subspace $S_X$ of ${\mathbf{C}}^{n-1}$ . If $\langle S_X,p_C\rangle _{\mathbf {R}}$ denotes the $\textbf {R}$ -affine span of the two affine subspaces of ${\mathbf{C}}^{n-1}$ , we will prove that the open dense subset

$$ \begin{align*} \pi^{-1}({\textbf{C}}^{n-1}\setminus \langle S_X,p_C\rangle_{\mathbf{R}}) \end{align*} $$

is contained in S.

Figure 1 The proof of Lemma 6.6.

Indeed, for any point y in ${\mathbf{C}}^{n-1}\ltimes \textbf {R}$ such that $\pi (y)$ does not belong to $\langle S_X,p_C\rangle _{\mathbf {R}}$ , the complex affine line determined by $\pi (y)$ and $p_C$ intersects $S_X$ at a unique point $z_y$ . The three points $(p_C, z_y, \pi (y))$ are not $\textbf {R}$ -colinear and determine a unique Euclidean circle $E_y$ . The unique chain $\mathcal C$ projecting to $E_y$ and containing y will, by construction, intersect C at a point c and X at a point x, which shows that $y\in S$ .

We can now conclude the proof of Proposition 6.2.

Proof of Proposition 6.2

We argue by induction.

In the case where $n=1$ , we know that for almost every positively oriented triple $(x,y,z)$ , the triple $(\phi (x),\phi (y),\phi (z))$ is contained in a $2p$ -dimensional linear subspace of the Hilbert space ${\mathcal H}$ (as in §2.2) of signature $(p,p)$ . By Fubini’s theorem, one can find $x,y$ such that $\phi (x)$ and $\phi (y)$ are opposite (that is, span a subspace of signature $(p,p)$ ) and for almost all z, $\phi (z)$ lies in the span $\langle \phi (x),\phi (y)\rangle $ .

For the inductive step, combining Lemmas 6.4 and 6.5, we deduce that the set $\mathcal A$ of pairs $(C,X)\in \mathcal F_0$ , such that the restriction of $\phi $ to X almost surely maps chains to chains and such that all conditions of Lemma 6.5 hold for $(C,X)$ , has full measure in $\mathcal F_0$ . In particular, $\mathcal A$ is not empty and we can chose a pair $(C,X)\in \mathcal {A}$ .

By the inductive hypothesis and Lemma 6.5(4), there is a $np$ -dimensional linear subspace $V_{p,(n-1)p}$ of ${\mathcal H}$ such that $\phi (C\cap X)< V_{p,(n-1)p}$ and for almost every $x\in X$ , $\phi (x)< V_{p,(n-1)p}$ . Let us choose a point y in C such that the pair $(y, C\cap X)$ is generic for $\phi $ and define

$$ \begin{align*}V_{p,np}=\langle V_{p,(n-1)p},\phi(y)\rangle.\end{align*} $$

Because the pair $(y, C\cap X)$ is generic for $\phi $ , for almost every point $c\in C$ , $\phi (c)< V_{p,np}$ . Because, by Lemma 6.5(2), almost every pair $(c,x)\in C\times X$ is generic, there exist a full measure subset of $S=\bigcup \mathcal C_{c,x}$ consisting of points s with $\phi (s)< V_{p,np}$ . The conclusion follows because, by Lemma 6.6, the set S contains an open dense subset of $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ and hence a full measure subset of S has full measure in $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ .

6.2 Rigidity of maximal representations of complex hyperbolic lattices

We now have all the needed ingredients to prove our rigidity result for maximal representations of complex hyperbolic lattices.

Theorem 6.7. Let $n\geq 2$ and let ${\Gamma }<\operatorname {\mathrm{SU}}(1,n)$ be a complex hyperbolic lattice, and let $\rho :\Gamma \to \operatorname {\mathrm{PO}}_{\mathbf{C}}(p,\infty )$ be a maximal representation. If there is a $\rho $ -equivariant measurable map $\phi \colon \partial {{\mathcal X}_{\mathbf{C}}(1,n)}\to \mathcal I_p$ , then there is a finite-dimensional totally geodesic Hermitian symmetric subspace $\mathcal Y\subset {\mathcal X}_{\mathbf{C}}(p,\infty )$ that is invariant by $\Gamma $ . Furthermore, the representation $\Gamma \to \operatorname {\mathrm{Isom}}(\mathcal Y)$ is maximal.

Proof. By Corollary 6.1 and Proposition 6.2, we know that the image of $\phi $ is essentially contained in the boundary of some ${\mathcal X}_{\mathbf{C}}(p,np)$ . Because $\Gamma $ is countable, we can find a $\Gamma $ -invariant full measure subset of $\partial {{\mathcal X}_{\mathbf{C}}(1,n)}$ whose image in contained in $\partial {\mathcal X}_{\mathbf{C}}(p,np)$ . In particular, this copy of ${\mathcal X}_{\mathbf{C}}(p,np)$ is $\Gamma $ -invariant. This concludes the proof.

Proof of Theorem 1.1

Under the hypothesis of Zariski density, a measurable $\rho $ -equivariant map $\phi : \partial {{\mathcal X}_{\mathbf{C}}(1,n)}\to \mathcal I_p$ is given by Theorem 1.7. If $p\leq 2$ , we know from Proposition 5.15 that the representation $\rho $ virtually splits as a product of geometrically dense maximal representations, and therefore is enough to understand the case in which $\rho $ is geometrically dense. In this case, the existence of a measurable $\rho $ -equivariant map $\phi \colon \partial {{\mathcal X}_{\mathbf{C}}(1,n)}\to \mathcal I_p$ is given by Corollary 4.17.

Remark 6.8. Combining the results of this paper and those of [KM17], one can deduce that if $\Gamma <\operatorname {\mathrm{SU}}(1,n)$ is cocompact and $\rho :\Gamma \to \operatorname {{\textrm O_{\mathbf{C}}}}(p,\infty )$ is maximal, then there is a totally geodesic subspace $\mathcal Y\subset {\mathcal X}_{\mathbf{C}}(p,\infty )$ preserved by $\rho (\Gamma )$ which is isometric to ${\mathcal X}_{\mathbf{C}}(1,n)\times {\mathcal X}_{\mathbf{C}}(p_1,\infty )\times \cdots \times {\mathcal X}_{\mathbf{C}}(p_k,\infty )$ after a suitable rescaling of the metric of the various factors, where $p_i>2$ . Furthermore, the induced action on ${\mathcal X}_{\mathbf{C}}(p_i,\infty )$ is maximal and geometrically dense, but not Zariski dense. If $\Gamma <\operatorname {\mathrm{SU}}(1,n)$ is non-uniform, we can deduce from [Poz15] the same result where possibly $\mathcal Y$ also has some finite-dimensional factors of tube type. We conjecture that indeed $\mathcal Y={\mathcal X}_{\mathbf{C}}(1,n)$ : the absence of factors of type ${\mathcal X}_{\mathbf{C}}(p_i,\infty )$ would be implied by a positive answer to Question 1.6.

7 Maximal representations in $\operatorname {\mathrm{PO}}_{\mathbf {R}}(2,\infty )$

In this section, we will construct the example of Theorem 1.2. Here, we focus on lattices ${\Gamma }_{\Sigma }<\text {PSL}(2,\textbf {R})=\operatorname {\textrm {PU}}(1,1)$ . A finite index subgroup of ${\Gamma }_{\Sigma }$ is then the fundamental group of a Riemann surface of negative Euler characteristic.

To construct geometrically dense maximal representations, we need to recall some of the geometry of ${\mathcal X}_{\mathbf {R}}(2,\infty )$ and of the specific boundary where the Bergmann cocycle is defined. Recall from §5.3 that the Bergmann cocycle $\beta _{\mathbf {R}}:\mathcal I_1(2,\infty )^3\to \{-2,0,2\}$ induces a $\operatorname {\textrm {O}}_{\mathbf {R}}^+(2,\infty )$ -invariant partial cyclic ordering on the set of isotropic lines $\mathcal I_1(2,\infty )$ : we say that $(x,y,z)$ is positively oriented if and only if $\beta _{\mathbf {R}}(x,y,z)=2$ . This is a consequence of the fact that $\beta _{\mathbf {R}}$ is a cocycle, and hence if $\beta _{\mathbf {R}}(x,y,z)=2$ and $\beta _{\mathbf {R}}(x,z,t)=2$ , then necessarily $\beta _{\mathbf {R}}(x,y,t)=2$ and $\beta _{\mathbf {R}}(y,z,t)=2$ . We say that a triple $(x,y,z)\in \mathcal I_1(2,\infty )$ is maximal if $\beta _{\mathbf {R}}(x,y,z)=2$ . It is easy to check that maximal triples form a single $\operatorname {\textrm {O}}^+_{\mathbf {R}}(2,\infty )$ -orbit. More generally, we say that an n-tuple $(x_1,\ldots ,x_n)$ is maximal if every subtriple $(x_i,x_j,x_k)$ , with $i\leq j\leq k$ , is. Furthermore, given an opposite pair $(x,z)\in \mathcal I_1(2,\infty )$ , we denote by $I_{x,z}$ the interval with endpoints $(x,z)$ :

$$ \begin{align*} I_{x,z}=\{y\in \mathcal I_1(2,\infty)|\;(x,y,z)\text{ is maximal}\}. \end{align*} $$

The following property of intervals will be useful.

Proposition 7.1. Let $x,y$ be a pair of opposite points in $\cal {I}_1(2,\infty )$ . The interval $I_{x,y}$ is homeomorphic to a bounded convex subspace of a Hilbert space.

Proof. Recall from §5.3 that given two opposite isotropic lines $x,y\in \mathcal I_1(2,\infty )$ of which we choose representatives $\overline x,\overline y$ such that $Q(\overline x,\overline y)<0$ , the interval $I_{x,y}$ consists of the isotropic subspaces

$$ \begin{align*}I_{x,y}=\{z\in\mathcal I_1(2,\infty)|\;Q(\overline x,\overline z)<0, Q(\overline y,\overline z)<0, \text{ and } \textbf{or} ([\overline x],[\overline y],[\overline z])=+\}.\end{align*} $$

Indeed the expression of the restriction of the quadratic form to the subspace $x,y,z$ is represented, with respect to that basis $\{\overline x, \overline y, \overline z\}$ by the matrix

$$ \begin{align*}\left(\begin{matrix} 0& Q(\overline x,\overline y)& Q(\overline x,\overline z)\\Q(\overline x,\overline y)&0&Q(\overline y,\overline z)\\Q(\overline x,\overline z)&Q(\overline y,\overline z)&0\end{matrix}\right),\end{align*} $$

whose determinant, $2Q(\overline x,\overline y)Q(\overline x,\overline z)Q(\overline y,\overline z)$ , is negative if and only if the signs of $Q(\overline x,\overline z)$ and $Q(\overline y,\overline z)$ are equal and can be chosen negative.

Without lost of generality, we can find a Hilbert basis $(e_i)_{i\in \mathbf{N}}$ which is orthogonal for Q, such that $Q(e_1)=Q(e_2)=1$ , $Q(e_i)=-1$ for $i\geq 3$ and such that $x,y$ have representatives $\overline {x}=e_1+e_3$ and $\overline {y}=-e_1+e_3$ . For $z\in \mathcal {I}_1(2,\infty )$ , let $\overline {z}$ be a representative of z, such that $\|\overline {z}\|=\sqrt {2}$ (here the norm $\|\cdot \|$ is computed with respect to the scalar product $\langle \ ,\ \rangle $ defining the Hilbert space ${\mathcal H}$ ). We can write $\overline {z}=u+v$ with $\|u\|=\|v\|=1$ , u in the span of $\{e_1,e_2\}$ , and v in the orthogonal of $\{e_1,e_2\}$ . If we write $u=u_1e_1+u_2e_2$ and $v=v_3e_3+v'$ with $v'\bot e_3$ , then the requirements $Q(\overline x,\overline z)<0$ and $Q(\overline y,\overline z )<0$ are both satisfied if and only if $v_3>|u_1|$ . Furthermore, in this case, $\textbf {or}([\overline x],[\overline z],[\overline y])=+$ if and only if $u_2>0$ . In particular, $\mathcal I_1(2,\infty )$ is homeomorphic to the pairs $(u_1,v')$ with $|v'|^2+|u_1|^2<1$ , which is a bounded convex subset of a Hilbert space.

In analogy with the finite-dimensional case, we say that an element $g\in \operatorname {{\textrm O_{\mathbf {R}}}}^+(2,\infty )$ is Shilov hyperbolic if it has an attractive line in $\mathcal I_1(2,\infty )$ or equivalently if g has a real eigenvalue $\unicode{x3bb} _1(g)$ of absolute value strictly bigger than one and multiplicity one (observe that g has at most two eigenvalues of absolute value bigger than one, and in this case, we denote by $\unicode{x3bb} _1(g)$ the eigenvalue with highest absolute value). If g is Shilov hyperbolic, we denote by $g^+\in \mathcal I_1(2,\infty )$ the eigenline corresponding to $\unicode{x3bb} _1(g)$ and by $g^-\in \mathcal I_1(2,\infty )$ the eigenline corresponding to $\unicode{x3bb} _1(g)^{-1}$ .

To carry out our construction of geometrically dense maximal representation, we will need the following result, which ensures a good nesting property of the images of intervals under the action of Shilov-hyperbolic elements.

Proposition 7.2. Given a Shilov-hyperbolic element g and a maximal 4-tuple $(x,y,z,t)\in \mathcal I_1(2,\infty )$ such that also $(x,y,g^+,z,t,g^-)$ is maximal, there exists $n\in \mathbf{N}$ such that $(y,g^nx,g^+,g^nt,z)$ is maximal as well.

Proof. As in the proof of Proposition 7.1, we fix a Hilbert basis of ${\mathcal H}$ such that $Q(e_1,e_1)=Q(e_2,e_2)=1$ and $Q(e_i,e_i)=-1$ for all $i\geq 3$ . Because the group $\operatorname {{\textrm O_{\mathbf {R}}}}(2,\infty )$ acts transitively on pairs of opposite isotropic lines, we can, without loss of generality, assume that $\overline g^+=e_1+e_3$ and $\overline g^-=-e_1+e_3$ . Because the 6-tuple $(x,y,g^+,z,t,g^-)$ is maximal, we can fix, for every $w\in \{x,y,z,t\}$ , a lift $\overline w$ of the form $(\cos \theta _w,\sin \theta _w, w_3,\ldots )$ with the additional requirements that $\sum _{i\geq 3} w_i^2=1$ (because w defines an isotropic line) and that $w_3>0$ . Furthermore, because the restriction of Q to $\langle w,g^+,g^-\rangle $ has signature $(2,1)$ , we deduce that $Q(\overline x, \overline g^+)$ and $Q(\overline x,\overline g^-)$ have the same sign, and, in particular, $w_3>|\!\cos \theta _w|$ . Finally, the maximality of the 6-tuple $(x,y,g^+,z,t,g^-)$ implies that

$$ \begin{align*} -\pi <\theta_x<\theta_y<0<\theta_z<\theta_t<\pi. \end{align*} $$

We decompose each vector $\overline w$ in the relevant eigenspaces for g: $\overline {w}=w^+\overline {g}^++w^-\overline {g}^-+w_0$ (where the vector $w_0$ is orthogonal to $\langle g^+,g^-\rangle $ ). Observe that $w^+=(w_3+\cos \theta _w)/2$ and $w^-=(w_3-\cos \theta _w)/2$ . Because we know that $w_3>|\!\cos \theta _w|$ , we deduce that $w^+\neq 0$ and because $\|g^n(w^+\overline {g}^+)\|/\|g^n(\overline {w}-w^+\overline {g}^+)\|\geq |\unicode{x3bb} _1(g)/\unicode{x3bb} _2(g)|^n$ , where $\unicode{x3bb} _2(g)$ is the second maximal eigenvalue (possibly of absolute value 1), we can find n big enough such that $\theta _y<\theta _{g^nx}<0<\theta _{g^nt}<\theta _z$ . Up to considering $g^2$ instead of g, we may assume that $\unicode{x3bb} _1(g)>0$ and because $w_3>0$ , we have $g^n\overline w/\|g^n\overline w\|\to \overline {g}^+/\|\overline {g}^+\|$ . So by continuity of Q, $Q(g^n\overline {w},y)$ has the same sign as $Q(\overline {g}^+,y)$ for n large enough.

Hence we can find n such that the restriction of Q to $\langle y,g^nx,g^+\rangle $ has signature $(2,1)$ and $\theta _y<\theta _{g^nx}<0$ , which implies that the orientation $\textbf {or} ([\overline y],[g^n \overline x], [\overline {g}^+])$ is positive. For such n, the triple $(y,g^nx,g^+)$ is maximal and similarly, we can also assume, up to possibly further enlarging n, that $(g^+,g^nt,z)$ is maximal for n large enough. Together with the fact that $(g^nx,g^+,g^nt)$ is maximal for any n, this is enough to guarantee that $(y,g^nx,g^+,g^nt,z)$ is maximal for n large enough.

Proposition 7.3. There exists a maximal 4-tuple $(x,y,z,t)\in \mathcal I_1(2,\infty )$ and Shilov- hyperbolic elements $g,h\in \operatorname {\mathrm{O}}_{\mathbf {R}}^+(2,\infty )$ such that the 8-tuple $(x,h^+,y,g^+,z,h^-,t,g^-)$ is maximal and such that the group generated by g and h does not preserve any finite-dimensional subspace of ${\mathcal H}$ .

Proof. We decompose the Hilbert space ${\mathcal H}$ as a direct sum ${\mathcal H}={\cal V}\oplus \cal W$ , where ${\cal V}$ and $\cal W$ are orthogonal with respect to Q, the restriction of Q to ${\cal V}$ has signature $(2,2)$ , and $({\mathcal W},-Q|_{\mathcal W})$ is a Hilbert space with Hilbert basis $(e_i)_{i\geq 3}$ . We choose an element $g\in \operatorname {\textrm {O}}_{\mathbf {R}}^+(2,\infty )$ that induces a Shilov-hyperbolic element of ${\cal V}$ , and acts on each subspace $\cal L_i:=\langle e_{2i+1},e_{2i+2}\rangle $ of ${\mathcal W}$ as a rotation of angle $\theta _i$ , where $\theta _i/\pi $ are distinct irrational numbers modulo two. Observe that the attractive (respectively repulsive) eigenlines $g^\pm $ of g belong to $\mathcal I_1({\cal V})\subset \mathcal I_1(2,\infty )$ . Furthermore, every invariant subspace for the g action is obtained as the direct sum of a subspace of ${\cal V}$ and a sum of the $\cal L_i$ .

We construct a basis $\{f_1,f_2,e_1,e_2\}$ of ${\cal V}$ which is orthogonal for Q and such that $Q(f_1,f_1)=Q(f_2,f_2)=1$ , so that $Q|_{\langle e_1,e_2\rangle }$ is negative definite. Let ${\mathcal W}_0={\mathcal W} \oplus \langle e_1,e_2\rangle $ (recall that the restriction of Q to $\cal W$ is negative definite). Choose two independent vectors v and $v'$ in ${\mathcal W}_0$ whose projection on every $\textbf {R} e_i$ (for $i\geq 1$ ) is different from 0. Let ${\cal V}'=\langle {\cal V},v,v'\rangle $ ; the restriction of Q to ${\cal V}'$ has signature $(2,4)$ . Because ${\cal V}'$ is finite-dimensional, we can choose $x,y,z,t\in \mathcal I_1({\cal V}')=\mathcal I_1(2,4)$ and an isometry $h_0\in \operatorname {\textrm {SO}}^+(2,4)$ such that the 8-tuple $(x,h_0^+,y,g^+,z,h_0^-,t,g^-)$ is maximal and there is no $h_0$ -invariant subspace of $\mathcal V'\subset {\mathcal H}$ that is invariant by g.

Let ${\mathcal W}'\subset {\mathcal W}$ be the orthogonal of $\mathcal V'$ , and choose a Hilbert basis of ${\mathcal W}'$ consisting of vectors which have a non-trivial projection on every $\textbf {R} e_i$ . We choose h that acts as the hyperbolic isometry $h_0$ of $\mathcal V'$ , and h acts on ${\mathcal W}'$ as g does on ${\mathcal W}$ . The group generated by g and h does not preserve any finite-dimensional subspace: because every subspace $\mathcal Z\subset {\mathcal H}$ , which is invariant by h, will be either contained in $\mathcal V'$ (and then it is trivial by assumption if it is also invariant by g) or contain a vector whose projection on every $\textbf {R} e_i$ is not trivial. However, then it must contain ${\mathcal W}_0$ , if it is g-invariant and therefore cannot be h-invariant.

Remark 7.4. If $g,h$ are constructed as in the proof of Proposition 7.3, for every integer n, the pair $g^n,h^n$ satisfies the conclusion of Proposition 7.3 as well.

Given an interval $I_{a,b}$ , we denote its closure by $\overline {I_{a,b}}$ for the quotient topology on the projective space $\textbf {P}{\mathcal H}$ coming from the Hilbert topology on ${\mathcal H}$ . The following property of intervals is also useful.

Proposition 7.5. Assume $(a,b,c,d)\in \mathcal I_1(2,\infty )^4$ is maximal, then $\overline {I_{b,c}}\subset I_{a,d}$ .

Proof. As above, we can assume without loss of generality that $\overline b=e_1+e_3$ , $\overline c=-e_1+e_3$ for a Hilbert basis orthogonal for Q, such that $Q(e_1)=Q(e_2)=1$ , $Q(e_i)=-1$ for $i\geq 3$ . A generic point $t\in \overline {I_{b,c}}$ will then have a representative of the form $\overline t=(\cos \theta _t,\sin \theta _t,v_t,w^t_1,\ldots )$ , where $w^t\in \langle e_4,\ldots \rangle $ , $\|w^t\|^2+v_t^2=1$ , $0\leq \theta _t\leq \pi $ , and $v_t\geq |\!\cos \theta _t|$ . A similar computation shows that the classes $a,d$ will have representatives $\overline a,\overline b$ of a similar form such that $\|w^a\|^2+v^2_a=\|w^d\|^2+v^2_d=1$ , $-\pi \leq \theta _d<\theta _a\leq 0$ and $v_a>|\!\cos \theta _a|$ , $v_d>|\!\cos \theta _d|$ .

To verify that $(a,t,d)$ is maximal, it is enough to verify that $Q(\overline a,\overline t)$ and $Q(\overline d,\overline t)$ are negative (the condition with the orientation follows immediately from the analog property for intervals in the circle): an explicit computation gives

$$ \begin{align*}Q(\overline a,\overline t)=\cos\theta_a\cos\theta_t-v_av_t +\sin\theta_a\sin\theta_t-\langle w_a,w_t\rangle<0.\end{align*} $$

More precisely, because $v_a>|\!\cos \theta _a|$ and $v_t\geq |\!\cos \theta _t|$ ,

(2) $$ \begin{align}\cos\theta_a\cos\theta_t-v_av_t\leq0.\end{align} $$

Furthermore, because $-\sin \theta _a>\| w_a\|$ and $\sin \theta _t\geq \| w_t\|$ ,

(3) $$ \begin{align}\sin\theta_a\sin\theta_t-\langle w_a,w_t\rangle<\sin\theta_a\sin\theta_t+\| w_a\|\| w_t\|\leq0.\end{align} $$

Observe that equations (2) and (3) cannot be an equality simultaneously because if equation (2) is an equality, then $\theta _t=\pi /2$ and thus $\sin (\theta _t)=1$ . The verification that $Q(\overline d,\overline t)<0$ is identical and thus the result follows.

Combining Propositions 7.2, 7.3, and 7.5, we obtain the following.

Corollary 7.6. There exists a maximal 4-tuple $(x,y,z,t)$ in $\mathcal I_1(2,\infty )$ , and a pair of Shilov-hyperbolic elements $A,B\in \operatorname {\mathrm{O}}_{\mathbf {R}}^+(2,\infty )$ that play ping-pong with this tuple, namely such that

We can furthermore assume that the group generated by $A,B$ does not leave invariant any finite-dimensional subspace of ${\mathcal H}$ .

Proof. Let $g,h$ be the Shilov-hyperbolic elements and $(x,y,z,t)$ be the points given by Proposition 7.3. Proposition 7.2 implies that we can find an integer n such that the pair $(A,B)=(h^n,g^n)$ plays ping-pong with the 4-tuple. Moreover, we can pass to the closure thanks to Proposition 7.5. The second claim is a consequence of Remark 7.4.

Proposition 7.7. Let $\Sigma $ be the once punctured torus, and let $a,b$ be the standard generators of ${\Gamma }_{\Sigma }=\pi _1(\Sigma )$ oriented as in the picture.

Assume that $\rho:{\Gamma}_{\Sigma}\to\operatorname{\mathrm{O}}_{\mathbf{R}}^+(2,\infty)$ has the property that the image $\rho(aba^{-1}b^{-1})$ has a fixed point l in $\mathcal I_1(2,\infty)$ . Then

$$ \begin{align*} 2i_\rho=\beta_{\mathbf{R}}(l,\rho(a^{-1})l,\rho(ba)^{-1}l)+\beta_{\mathbf{R}}(\rho(ba)^{-1}l,\rho(b^{-1})l,l). \end{align*} $$

Proof. As the (relative) bounded cohomology of a surface with a puncture and that of a homotopic surface with a boundary component are canonically isomorphic, we can realize $\Sigma $ as a surface with geodesic boundary $\partial \Sigma $ . We denote by $\text {H}_b^2(\Sigma ,\textbf {R})$ the singular bounded cohomology of the topological space $\Sigma $ (namely the cohomology of the complex of bounded singular cochain), and by $\text { H}_b^2(\Sigma ,\partial \Sigma ,\textbf {R})$ the relative bounded cohomology, which is the cohomology of the complex of bounded cochains that vanishes on singular simplices with image entirely contained in $\partial \Sigma $ .

It follows from [BIW10, Theorem 3.3] that the Toledo invariant $i_\rho $ can be computed from the formula

$$ \begin{align*}2i_\rho=\langle j_{\partial\Sigma}^{-1}g_{\Sigma}\rho^*(\kappa_b),[\Sigma,\partial\Sigma]\rangle.\end{align*} $$

Here, $g_{\Sigma }:\text {H}_b^2({\Gamma }_{\Sigma },\textbf {R})\to \text {H}_b^2(\Sigma ,\textbf {R})$ is the canonical isomorphism and $j_{\partial \Sigma }^{-1}:\text {H}_b^2(\Sigma ,\textbf {R})\to \text {H}_b^2(\Sigma ,\partial \Sigma ,\textbf {R})$ is the isometric isomorphism described in [BBF⁺14] that is inverse to the map induced by the inclusion of bounded relative cochains in bounded cochains. Recall that, whenever a base point $x\in \widetilde \Sigma $ , the universal cover, is fixed, the bounded cohomology $\text { H}_b^2(\Sigma ,\textbf {R})$ can be also isometrically computed from the complex of functions on straight simplices with vertices in the set ${\Gamma }_{\Sigma }\cdot x\subset \widetilde \Sigma $ . Furthermore, if c is a cocycle representing the class $[c]\in \text { H}_b^2(\Gamma _{\Sigma },\textbf {R})$ , the class $g_{\Sigma }([c])$ is represented by the cocycle

$$ \begin{align*}\overline c(\Delta(g_0x,g_1x,g_2x))=c(g_0,g_1,g_2).\end{align*} $$

We denote, as in Lemma 5.4, $C_{\beta }^l\in C^2_b(\Gamma _{\Sigma },\textbf {R})$ the cocycle defined by

$$ \begin{align*}C_{\beta}^l(g_0,g_1,g_2)=\beta_{\mathbf{R}}(\rho(g_0)l,\rho(g_1)l,\rho(g_2)l)\end{align*} $$

(recall that $l\in \mathcal I_1(2,\infty )$ is a fixed point of $\rho (bab^{-1}a^{-1}))$ ). We deduce that $\overline C_{\beta }^l$ vanishes on simplices contained in $\partial \Sigma $ , as long as we choose $x\in \widetilde \Sigma $ in the preimage of $\partial \Sigma $ . Thus,

$$ \begin{align*}\langle j_{\partial\Sigma}^{-1}g_{\Sigma}\rho^*(\kappa_b),[\Sigma,\partial\Sigma]\rangle=\sum_i a_i\beta_{\mathbf{R}}(g^i_0l,g^i_1l,g^i_2l),\end{align*} $$

provided $\sum _i a_i\Delta (g^i_0x,g^i_1x,g^i_2x)$ represents the relative fundamental class $[\Sigma ,\partial \Sigma ]$ .

Observe that a relative fundamental class for the once punctured torus can be written as the sum of the triangles $\Delta (x,a^{-1}x,b^{-1}a^{-1}x)$ , $\Delta (x, b^{-1}a^{-1}x,a^{-1}b^{-1}x)$ , and $\Delta (a^{-1}b^{-1}x,b^{-1}x,x)$ , and the cocycle $\beta _{\mathbf {R}}$ vanishes on the third simplex because $\beta _{\mathbf {R}}$ is ${\Gamma }_{\Sigma }$ -equivariant and alternating, and $\Delta (x, b^{-1}a^{-1}x,a^{-1}b^{-1}x)=\Delta (abx, x,[a,b]x)$ . The result follows.

Proof of Theorem 1.2

Let $A,B\in \operatorname {\textrm {O}}_{\mathbf {R}}^+(2,\infty )$ as given by Corollary 7.6. The group $\Gamma _{\Sigma }$ is a free group on two generators a and b. We define the representation $\rho $ by setting $\rho (a)=A$ , $\rho (b)=B$ . Corollary 7.6 implies that $\rho (bab^{-1}a^{-1})I_{y,z}\subset I_{y,B^+}$ .

Because the interval $I_{y,z}$ is a non-empty bounded convex set of a Hilbert space (Proposition 7.1) whose closure is compact in the weak topology, we deduce using Tychonoff’s fixed point theorem [Tyc35] (see [DS88] for a modern proof) that the continuous function $\rho ([a,b]): \overline {I_{y,z}}\to \overline {I_{y,z}}$ has a fixed point.

Because l belongs to the interval $\overline {I_{y,z}}\subset \overline {I_{y,x}}$ , we have that $\rho (a^{-1})l$ belongs to the interval $I_{z,t}$ and $\rho (a^{-1}b^{-1})l=\rho (b^{-1}a^{-1})l$ belongs to the interval $I_{t,x}$ . This implies that

$$ \begin{align*}\beta_{\mathbf{R}}(l,\rho(a^{-1})l,\rho(ba)^{-1}l)=2;\end{align*} $$

the verification that $\beta _{\mathbf {R}}(\rho (ba)^{-1}l,\rho (b^{-1})l,l)=2$ is analogous. Together with Proposition 7.7, this shows that $i_\rho =2$ , namely that the representation $\rho $ is maximal.

We conclude the proof verifying that the representation is geometrically dense. Because the representation is irreducible, there is no fixed point at infinity and thus there is a minimal totally geodesic invariant subspace, which cannot be of finite dimension because of Lemma 3.11. It has no Euclidean factor otherwise there would be a fixed point at infinity or a pair of such fixed points, which is impossible thanks to Proposition 2.4. So either it is of rank one, a product of two rank-1 subspaces, or a rank-2 subspace. Lemma 5.14 excludes the presence of rank-1 factors and that the symmetric subspace is associated to $\operatorname {{\textrm O_{\mathbf {H}}}}(2,\infty )$ . The closure cannot be associated to $\operatorname {{\textrm O_{\mathbf{C}}}}(2,\infty )$ by Theorem 1.1. Therefore, the minimal totally geodesic subspace is isometric to the symmetric subspace to $\operatorname {{\textrm O_{\mathbf {R}}}}(2,\infty )$ . By possibly restricting to the isometry group of that subspace, we can assume that the representation is geometrically dense.