Radio astronomy has provided important surveys that have made possible key (and sometimes serendipitous) discoveries. I will briefly mention some of the past continuum and line (HI) radio surveys as well as new, on-going surveys and surveys planned for the near future. This new generation of large radio surveys is bringing extra challenges in terms of data handling but also great new possibilities thanks to the wider range of data products that they will provide.

We review the history of X-ray sky surveys from the early experiments to the catalogues of 105 sources produced by ROSAT, Chandra and XMM-Newton. At bright fluxes the X-ray sky is shared between stars, accreting binaries and extragalactic sources while deeper surveys are dominated by AGN and clusters of galaxies. The X-ray background, found by the earliest missions, has been largely resolved into discrete sources at soft (0.3-2 keV) energies but at higher energies an important fraction still escapes detection. The possible identification of the missing flux with Compton-thick AGN has been probed in recent years by Swift and Integral.

Variability seen in objects observed at different epochs has proved to be an excellent discriminator for rare classes of objects. The comparison of ROSAT All Sky Survey (RASS) and ROSAT pointed observations identified several Novae and high variability AGN as well as initiating the observational study of Tidal Disruption events. More recently the XMM-Newton slew survey, in conjunction with archival RASS data, has detected further examples of flaring objects which have been followed-up in near-real time at other wavelengths.

LAMOST is a special reflecting Schmidt telescope used to observe ten million spectra of celestial objects. There are about half million spectra released in the pilot survey of LAMOST. In the “Big Data“ era of astronomy, Virtual Observatory will play an important role to make use of those massive spectral data of LAMOST.

The Atacama Large Millimeter/submillimeter Array (ALMA) is nearing its phase of full operations. ALMA will collect about 200TB/year of astronomical data which will be reduced by an automatic pipeline and turned into fully calibrated science-ready data products. We present design choices, challenges and solutions from data capturing over data reduction and data distribution to archival research, that allow to deal with the large amounts of data and, hopefully, achieve the maximum amount of science return.

The Large Synoptic Survey Telescope (LSST; Ivezic et al.2008, http://lsst.org) is a planned, large-aperture, wide-field, ground-based telescope that will survey half the sky every few nights in six optical bands from 320 to 1050 nm. It will explore a wide range of astrophysical questions, ranging from discovering killer asteroids, to examining the nature of dark energy. LSST will produce on average 15 terabytes of data per night, yielding an (uncompressed) data set of 200 petabytes at the end of its 10-year mission. Dedicated HPC facilities (with a total of 320 TFLOPS at start, scaling up to 1.7 PFLOPS by the end) will process the image data in near real time, with full-dataset reprocessing on annual scale. The nature, quality, and volume of LSST data will be unprecedented, so the data system design requires petascale storage, terascale computing, and gigascale communications.

Advances in both digital processing devices and in technologies to sample the focal and aperture planes of radio antennas is enabling observations of the radio sky with high spectral and spatial resolution combined with large bandwidth and field of view. As a consequence, survey mode radio astronomy generating vast amounts of data and involving globally distributed collaborations is fast becoming a primary tool for scientific advance. The Square Kilometre Array (SKA) will open up a new frontier in data intensive astronomy. Within the next few years SKA precursor telescopes will demonstrate new technologies and take the first major steps toward the SKA. Projects that path find the scientific journey to the SKA with these and other telescopes are currently underway and being planned. The associated exponential growth in data require us to explore new methodologies for collaborative end-to-end execution of data intensive observing programs.

As we move ever closer to the Square Kilometre Array era, support for real-time, interactive visualisation and analysis of tera-scale (and beyond) data cubes will be crucial for on-going knowledge discovery. However, the data-on-the-desktop approach to analysis and visualisation that most astronomers are comfortable with will no longer be feasible: tera-scale data volumes exceed the memory and processing capabilities of standard desktop computing environments. Instead, there will be an increasing need for astronomers to utilise remote high performance computing (HPC) resources. In recent years, the graphics processing unit (GPU) has emerged as a credible, low cost option for HPC. A growing number of supercomputing centres are now investing heavily in GPU technologies to provide O(100) Teraflop/s processing. I describe how a GPU-powered computing cluster allows us to overcome the analysis and visualisation challenges of tera-scale data. With a GPU-based architecture, we have moved the bottleneck from processing-limited to bandwidth-limited, achieving exceptional real-time performance for common visualisation and data analysis tasks.

The massive amount of data produced by the recent multi-wavelength large-area surveys has spurred the growth of unprecedentedly massive and complex astronomical datasets that are proving the traditional data analysis techniques more and more inadequate. Knowledge discovery techniques, while relatively new to astronomy, have been successfully applied in several other quantitative disciplines for the determination of patterns in extremely complex datasets. The concerted use of different unsupervised and supervised machine learning techniques, in particular, can be a powerful approach to answer specific questions involving high-dimensional datasets and degenerate observables. In this paper I will present CLaSPS, a data-driven methodology for the discovery of patterns in high-dimensional astronomical datasets based on the combination of clustering techniques and pattern recognition algorithms. I shall also describe the result of the application of CLaSPS to a sample of a peculiar class of AGNs, the blazars.

In this work, we propose the use of Kernel Principal Component Analysis (KPCA) combined with k = 1 nearest neighbor algorithm (1NN) as a framework for supernovae (SNe) photometric classification. It is specially recommended for analysis where the user is interested in high purity in the final SNe Ia sample. Our method provide good purity results in all data sample analyzed, when SNR⩾5. As a consequence, we can state that if a sample as the Supernova Photometric Classification Challenge were available today, we would be able to classify ≈ 15% of the initial data set with purity higher than 90%. This makes our algorithm ideal for a first approach to an unlabeled data set or to be used as a complement in increasing the training sample for other algorithms. Results are sensitive to the information contained in each light curve, as a consequence, higher quality data (low noise) leads to higher successful classification rates.

The Virtual Atomic and Molecular Data Centre (VAMDC, http://www.vamdc.eu) is an international Consortium that has created an interoperable e-science infrastructure for the exchange of atomic and molecular data. The VAMDC defines standards for the exchange of atomic and molecular data, develop reference implementation of those standards, deploys registries of internet resources (yellow pages), designs user applications in order to meet the user needs, builds data access layers above databases to provide unified outputs from these databases, cares about asynchronous queries with workflows and connects its infrastructure to the grid. The paper describes the current service deployment of the VAMDC data infrastructure across our registered databases and the key features of the current infrastructure.

Extensive discoveries of variable stars in ground-based photometric surveys and in observations from space missions that provide unprecedented accuracy demonstrate that, till the most recent time, we knew only a tiny fraction of all detectable variable stars of our Galaxy. As a result, our knowledge on stellar variability types, related physical processes, variable-star statistics turn out to be based on an unrepresentative sample and are expected to be radically revised in the near future. The flow of new discoveries results in quite difficult problems for catalogs of variable stars, making it impossible to proceed in their compilation in the traditional way. Regretfully, automatic solutions are still not completely satisfying. Though not able to suggest a perfect way out, we present our vision of the future of variable-star research.

Galaxy Zoo (www.galaxyzoo.org) is familiar to many as a hugely successful public engagement project. Hundreds of thousands of members of the public have contributed to Galaxy Zoo which collects visual classifications of galaxies in Sloan Digital Sky Survey and Hubble Space Telescope images. Galaxy Zoo has inspired a suite of similar Citizen Science projects known as “The Zooniverse“ (www.zooniverse.org) which now has well over half a million participants. Galaxy Zoo has also shown itself, in a series of peer reviewed papers, to be a fantastic database for the study of galaxy evolution. In this invited talk I described how that public engagement via citizen science is not only an effective means of outreach from data intensive surveys, but if done right can and must also increase the scientific output of the survey.

The fast growing number of science data repositories is opening enormous possibilities to scientists all over the world. The emergence of citizen science projects is engaging in science discovery a large number of citizens globally. Astronomical research is now a possibility to anyone having a computer and some form of data access. This opens a very interesting and strategic possibility to engage large audiences in the making and understanding of science. On another perspective it would be only natural to imagine that soon enough data mining will be an active part of the academic path of university or even secondary schools students. The possibility is very exciting but the road not very promising. Even in the most developed nations, where all schools are equipped with modern ICT facilities the use of such possibilities is still a very rare episode. The Galileo Teacher Training Program GTTP, a legacy of IYA2009, is participating in some of the most emblematic projects funded by the European Commission and targeting modern tools, resources and methodologies for science teaching. One of this projects is Discover the Cosmos which is aiming to target this issue by empowering educators with the necessary skills to embark on this innovative path: teaching science while doing science.

There are several outstanding mysteries in interstellar medium spectroscopy which have remained unsolved after decades of effort. The diffuse interstellar bands (DIBs) have been known for almost a century (Heger 1922). Although more than 400 bands from the near UV to near infrared have been detected, none of them has been identified. In the Milky Way Galaxy, DIBs have been seen towards over one hundred stars. In the Magellanic Clouds, DIBs have been seen in the spectrum of SN 1987A as well as in the spectra of reddened stars (Ehrenfreund et al. 2002). DIB carriers in the interstellar medium of external galaxies can be probed by supernovae (Sollerman et al. 2005), and DIBs have been detected in external galaxies with redshifts up to 0.5 (Sarre 2006).

We present an overview of the present observational status of unexplained spectral phenomena in the ISM. The possibility of organic molecules and solids as the carrier of the DIB, 217 nm feature, ERE, UIR, and the 21 and 30 μm features is discussed.

When referring to unidentified infrared emission features, one has in mind the series of aromatic IR bands (AIBs) between 3.3 and 15 μm that are observed in emission in many environments where UV photons irradiate interstellar matter. These bands are now used by astronomers to classify objects and characterize local physical conditions. However, a deep analysis cannot proceed without understanding the properties of the band carriers. Large polycyclic aromatic hydrocarbon molecules are attractive candidates but interstellar species are still poorly characterized. Various studies emphasize the need for tackling the link between molecular aromatic species, aliphatic material and very small carbonaceous grains. Other unidentified emission features such as the 6.9, 21 and 30 μm bands could be involved in the evolutionary scenario.

Some characteristics of the 21 and 30 μm features observed in carbon star dust shells are briefly discussed.

The near-infrared (NIR) spectral range (2–5 μm) contains a number of interesting features for the study of the interstellar medium. In particular, the aromatic and aliphatic components in carbonaceous dust can be investigated most efficiently with the NIR spectroscopy. We analyze NIR spectra of the diffuse Galactic emission taken with the Infrared Camera onboard AKARI and find that the aliphatic to aromatic emission band ratio decreases toward the ionized gas, which suggests processing of the band carriers in the ionized region.

In recent years, the fullerene species C60 (and to a lesser extent also C70) has been reported in the mid-IR spectra of various astronomical objects. Cosmic fullerenes form in the circumstellar material of evolved stars, and survive in the interstellar medium (ISM). It is not entirely clear how they form or what their excitation mechanism is.

Hydrogenated amorphous carbon materials, a-C(:H), whose optical properties evolve in response to UV irradiation processing are promising candidate materials for cosmic carbonaceous dust. The optical properties of a:C(:H) particles have been derived as a function of size, band gap and hydrogen content over a wide wavelength range (EUV-cm) and can be used to investigate the size-dependent evolution of a-C(:H) material properties in the ISM.