NUI Galway has joined an international consortium attempting to build 'data mining' systems to search millions of bytes of stored astronomical data. Dick Ahlstrom reports
You need a special kind of computer system to dig through millions and millions of bytes of data. This is the challenge facing an international research group which plans to build the world's first "virtual space observatory" that will mine existing data for new discoveries.
A team of researchers from the Departments of Information Technology and Experimental Physics at NUI Galway, led by Dr Aaron Golden and Dr Ray Butler, are to join this international effort. The European Space Agency's ST-ECF (Space Telescope-European Coordinating Facility) is organising the EU-funded project, known as "Astrovirtel".
Literally every second, somewhere on the Earth's surface or up in orbit, astronomical telescopes are making fresh observations and capturing new data. It gets stored as visual pictures, infrared and radio frequency measurements, X-ray and gamma ray observations and in many other forms.
The various observations, however, don't really connect with one another. The group making the observation can study each separately but more could be done with the data if it could be pooled in an effective way.
This is the idea behind Astrovirtel, explains Dr Golden, who will provide information technology expertise to the NUI Galway team. It will allow European scientists "to search within and analyse the enormous combined global astronomical data archives for a wide range of new and as-yet undiscovered scientific phenomena," he says.
Astrovirtel will be based at the European Southern Observatory headquarters in Garching, Germany. A pool of data, 11,000,000,000,000 bytes of it collected over the past 25 years, sits waiting to be explored.
"What we propose to do is build a search engine," says Golden. "Not only will the search engine go through 11 terabytes of spectral data, we want the software to analyse it in real time."
This sounds deceptively simple but the difficulties are real. Data is held in a variety of formats depending on how the observations were taken. They come from many different sources. And while different observations of the same objects have been made over the years, the data as it now exists doesn't connect.
The search engine will get around these problems, says Golden, "in a way very similar to the way Google and other search engines do it". It should allow an astrophysicist working anywhere in the world with access to the Garching database to make fresh discoveries using the old data. The overall project is unusual in that Astrovirtel doesn't start with a working search engine- it is being built in pieces, with contributors such as NUI Galway adding their specific part. "The science is really driving the technology," says Golden.
He and Ray Butler are interested in objects known as "brown dwarves", bodies bigger than Jupiter but smaller than our sun. They are hot but never reached a size big enough to support the fusion that makes the sun shine. "They are strange objects, not quite planets and not quite stars," says Golden. They have coronal activity and a photosphere like the sun but have weather like a planet. The Galway team are building tools to allow the archive to be mined for information about brown dwarves and then for this information to be fed through an information "pipeline" which will automatically align all of the various observations to be delivered as an "astrometric" solution and a "photometric" solution.
Pooling the data in this way should give astrophysicists a great deal more information about brown dwarf weather, luminosity and star-like flaring, says Golden. The same tools will also be useful for looking at other objects, for example galaxies.
The Galway team is just one small component of a comprehensive mining system that will transform the vast archive into a goldmine of information. Joining the project has given Irish scientists access to a data source that is usually closed to us. The Government has so far not joined the European Southern Observatory and without membership we can't get access to the data. Golden and Butler's involvement, however, has opened the doors. "We are the only Irish scientists who have access to this archive."