Maintaining access to excellent data and computing infrastructure is a fundamental requirement for world class astronomy research. Investment in data and computing resources will enable Australian astronomers to fully exploit the wealth of data coming from major astronomical projects, currently underway both here and overseas.
AAL’s High Performance Computing (HPC) Project targets a priority in the Australian Astronomy Decadal Plan (2016 – 2025) of world-class HPC and software capability for large theoretical simulations, and resources to enable processing and delivery of large data sets from these facilities. AAL uses NCRIS funding to support this priority, and the data and computing needs of Australian astronomers via a variety of programs/facilities:
Aided by AAL HPC allocations, a new survey is collecting data that covers the entire southern sky at low radio frequencies in search of undiscovered pulsars. The Southern-sky MWA Rapid Two-Metre (SMART) survey team are using the Swinburne University of Technology OzSTAR Supercomputer to crunch their data, via processing time awarded by AAL’s Astronomy Supercomputing Time Allocation Committee (ASTAC). Over the lifetime of the project (upwards of five years), the researchers will analyse well over three petabytes of data and expect to find hundreds of pulsars, as well as provide measurements of previously known pulsars in a frequency range rarely explored.
The SMART survey team are hunting for pulsars via the low-frequency Square Kilometre Array (SKA) precursor telescope, the Murchison Widefield Array (MWA), located at Inyarrimanha Ilgari Bundara, the CSIRO Murchison Radio-astronomy Observatory.
For more information on the SMART pulsar survey, please see this story under MWA project highlights.
ASTAC was formed by AAL and the ASTAC process is managed by ADACS. ASTAC allocates available time on the following supercomputer resources for dedicated national astronomy access:
AAL will consider ASTAC allocating time on any other supercomputer resources that have dedicated national astronomy access, where appropriate. There are usually two calls for proposals per year, announced around May and November, and closing in June and December. Please see more information on AAL’s ASTAC webpage.
Spectra is a good example of a complex astronomy data product, where light from an object is spilt into its component wavelengths to be analysed. For complex data products (like spectra) to be accessible to users, they must first be ingested or copied into the database system used by Data Central. This requires identifying objects to be created (essentially a bucket to hold a spectrum with some identifying metadata) and the spectrum objects themselves to be created (linked to these identifying objects). Previously, both the identifying objects and spectra objects were created essentially one-by-one. Even for relatively modest sized datasets, with 39,314 spectra, this would be a very slow process taking several hours. The Data Central team successfully managed to reduce ingestion time by fully parallelising the process and making other efficiency gains. This allowed more powerful systems with multiple CPUs to dramatically speed up ingestion times e.g. the ingestion of 39,314 spectra in only 22 min 20 sec, compared to 228 min 52 sec previously. With the ingestion times sped up by more than a factor of 10, the team are now in a position to provide much faster feedback to survey teams wishing to host their data with Data Central.
In December 2022, representatives from AAO Macquarie (Data Central) and Astronomy Data and Computing Services (ADACS) travelled to join an AAL delegation in Canberra at the Department of Education (DE). For two hours, the group presented to senior DE representatives about the capabilities within NCRIS-funded astronomy data and computing activities, and their application to the broader scientific community. This was a very successful presentation with great engagement from DE who were keen to understand the applicability to the wider data environment which is evolving at an ever-increasing pace.