In the study of the spectra of algebras of holomorphic functions on a Banach space E, the bidual E″ has a central role, and the spectrum is often shown to be locally homeomorphic to E″. In this paper we consider the problem of spectra of subalgebras invariant under the action of a group (functions f such that f ○ g = f). It is natural to attempt a characterization in terms of the space of orbits E″/~ obtained from E″ through the action of the group, so we pursue this approach here and introduce an analytic structure on the spectrum in some situations. In other situations we encounter some obstacles: in some cases, the lack of structure of E″/~ itself; in others, problems of weak continuity and non-approximability of functions in the algebra. We also define a convolution operation related to the spectrum.