As the cloud grows more ubiquitous, and many archives of the digital age cease to exist outside of the server, we need to take better stock of how we are preserving the records of the modern world for posterity

. . . The minister looks puzzled and asks what an official reply means.

Official: “It just says the Minister has asked me to thank you for your letter and we say something like ‘the matter is under consideration’ or, even if we feel so inclined, ‘under active consideration’.”

Minister: “What’s the difference?”

Official: Well, under consideration means we’ve lost the file, under active consideration means we’re trying to find it.”

– From the TV sitcom Yes Minister, scripted by Antony Jay and Jonathan Lynn

BUT WHAT if all the files were lost? Not in a metal filing cabinet, but in cyberspace, or the cloud, wherever that is.

Was the outcome of the 2012 Irish presidential election affected by a communication with no objective existence – a “tweet” conveying information which turned out to be false. We don’t know. What we do know though is that electronic communications technology has the potential to disrupt the democratic process, and all our elaborate record-keeping apparatus has not kept pace with new ways of communicating with each other.

In the digital age, keeping records has largely become easier. Many are just a click away on a “recent documents” tab or keyword search. But now that many important documents have no objective existence except as an electronic series of zeros and ones, we need to stop and think about what that means for the way we live. Will those files exist in 20, 30, 40 years when we need them? Or will they be stored on a floppy disk or piece of tape for which we have neither hardware nor software.

What if a 22nd-century scholar wishes to consult the Book of Kells in a virtual library? Will a click just bring up the ubiquitous 404 message – file not found?

Well-run businesses tend to have good information-technology practices. Systems are updated and archive material is copied on to new storage media. So do government departments and local authorities. But what of those which don’t? As history speeds up, major undertakings appear, flourish, and inevitably some crash and burn. With so much of the traditional public service privatised or outsourced, the likelihood of this generation passing on the “family records” to the next becomes more remote. And what if organisation A has a perfectly good database, and department B has another one, but they are not compatible?

We know about the big problems. Nasa has admitted to data corruption problems on space probe records. Valuable information won at great personal risk may be irretrievably lost. The United States is trying to recover data lost from a 1960 census. What of the worker trying to prove entitlement to benefit or pension based on service, when the dusty ledgers have disappeared?

Add to this the speeded-up redundancy in software and hardware used to keep records.

Think precious family photographs, once held in wallets of negatives and curling prints, then on floppy disks and CDs, unreadable by today’s tablets and iPads, perhaps on a remote website, sliding away into cyberspace when the one member of the family who kept all the family memorabilia dies or allows a subscription to lapse. Think of the proliferation of software and apps, supported now, superseded tomorrow.

Apply that on the national and international scale. The National Archives holds the records of the modern Irish State documenting our historical evolution. Operating under the aegis of the Department of Arts, Heritage and the Gaeltacht, it preserves records – mainly but not exclusively from government bodies – and makes them available to the public

According to Tom Quinlan, keeper at the National Archives in Dublin, some government departments have been exemplary in maintaining good records, minuting decisions, phone calls etc, and storing records which provide evidence of decision-making. The Departments of the Taoiseach and of Foreign Affairs have shown, in records released thus far, keen awareness of their historical responsibility.

“The problem is that in a world increasingly relying on email for official communications, we don’t have a designated system for preserving and retaining selected records.”

The idea of a central registry in which records can be classified, managed and preserved in a uniform manner has broken down, particularly with so many people generating material outside traditional structures, and in relatively novel formats, he says. “The advice used to be to ‘print to file’, but how realistic is that with video, websites and texts?”

And, as most public records are not released until 30 years after they are created, we have yet to see how digital record keeping will sit with paper-based records. (The most recent release covered 1981 and told us much about the behind-the-scene efforts to deal with the republican hunger strikes in which 10 men died that year.) Nor is it clear in what format documents have been stored and how accessible they will be.

Quinlan’s colleague, archivist Micheál Ó Conaire, points to another problem. “Documents need to be saved in a format which respects the original. For example, where bold or italic type is used to convey a specific meaning, this formating must be preserved or something irreplaceable may be lost.” For example, lawyers often use different colours for type or underlining to denote the status of the material thus indicated, but we don’t know if these distinctions will be evident in the archived version.

The Commission of Inquiry into the Dublin and Monaghan bombing final report published in March 2007 dealt at length about shortcomings in the way in which public information is stored. The then director of the National Archives, Dr David Craig, said that some government departments did not know what documents they held and others chose not to know – because if they did, they might have to field awkward questions about them.

Describing what happens in advance of the transfer of documents from a department to the National Archives for release under the 30-year rule, he said: “A major headache for the staff in some departments when they were being asked to prepare records of transfer, that they do not have any master information to start off with, so that they have to prowl around the basements and talk to staff in particular sections who might have records in their filing cabinets or whatever to find out what there is and so that they can prepare for the transfer. There is no central control in nearly all departments.”

Dealing with documents which were classified as secret, such as those held by the Garda Síochána, the experience of the National Archives was that, not infrequently, persons in control of limited collections of records of this type “. . . can find anything they want when they need it, but they don’t really want anybody else to be able to find it”.

Dr Craig felt that “people may, in a more genuine way, think that to let other people in the organisation know that certain records exist will undermine the confidentiality of that information and thereby allow the information to become available.” He submitted that even if the contents of files had to remain secret, for good reason, the existence of the files and the manner of their storage should be transparent.

Not surprisingly, the commission report, written by eminent senior counsel Patrick McEntee, found that large swathes of material from Garda files had gone missing. And the lack of a central registry was a huge problem.

To adapt former US defence secretary Donald Rumsfeld’s celebrated observation – how do we know what we know when we don’t even know what we have? Cue then for government, even at this late stage, to regulate the way in which official documents are preserved digitally.

The Department of Arts, Heritage and the Gaeltacht is responsible for policy on national archives, but the National Archives Advisory Council also has a role on advising on policy. Confusingly, the Department of Finance has the power to issue regulations to the public service on the way documents are stored, but despite requests from the National Archives, most recently in November 2011, has not done so.

Privately, officials admit that Ireland needs a digital repository with skilled people to staff it, but the collapse of State finances and a moratorium on public service appointments suggests that little is likely to happen in the immediate future. Last year, the National Library initiated a scheme whereby websites relating to the general election were archived. A National Archives pilot scheme in 2003 and 2004 to archive websites of government departments has not gone any further.

But if Ireland is standing still, the rest of the world is not. The British National Archives and Harvard University in Massachusetts have pooled efforts in the Unified Digital Formats Registry to bring together and share information among those responsible for managing format-critical preservation. That task is complicated by US legislation making it a criminal offence to circumvent the anti-piracy digital rights management software that manufacturers attach to software.

Not for the first time, the openness and universality which lies at the heart of the worldwide web collides with more traditional approaches to copyright.

The British National Archives head of digital preservation, Tim Gollins, has contested much of the “received wisdom” in the digital preservation community which says that imminent technological (software/data format) obsolescence is the primary threat.

He challenges the belief that the only way forward is a technically complex, expensive and difficult to operate integrated digital preservation system. Much of the software required is free or open source. His net point is that a good solution is not necessarily a very costly one.

Not all the problems are technical. Historian Jonathan Bardon believes that governments should instruct civil servants not to use personal email accounts for official communications, as to do so will reduce the completeness of the official record.

The British National Archives plans to reduce the 30-year rule to 20 years, by releasing two years’ worth of records from 2013 on, until in 2023, the 1993 papers will be available. Ireland will need to consider the effect of allowing the British version of Anglo- Irish affairs a free run, among other subjects. History is not best served by partial disclosure and staggered release favours first out of the blocks.

A nation as rich in history and culture as Ireland must take better care of its precious records. Issuing standard operating procedure guidelines to public bodies on storing data now is a low-cost and important step in that direction. Sometimes we need to look backwards to go forwards.

“It is possible to remove everything of significance from a file released under the 30- year rule by saying that it is complete except for a small number of secret documents . . .”

(Advice from the civil service, in Yes Minister)

Kieran Fagan edits the Irish Times coverage of the annual release of State papers under the 30-year rule in Belfast, Dublin and London

HISTORIAN JOHN BOWMAN: HOW WILL HISTORIANS OF THE FUTURE ACCESS AND INTERPRET RECORDS WHICH WILL NEVER HAVE EXISTED IN HARD COPY?

THE DIGITAL revolution has already transformed how historians work: much of it for the better with increased access to original paper records which have now been digitised. But what's coming down the track is much more complicated. How will historians of the future access and interpret records which will never have existed in hard-copy format?

Civil servants are enjoined to save the planet by not printing electronic documents. How will such documents be formally accessed by a government department? Who will decide their selection, preservation – or destruction – and their transfer to the National Archives?

And how will historians understand them? Word search can be powerful but lacks discrimination. Historians dealing with the early 20th century have the comfort that much of their basic data is handwritten letters: those dealing with the early 21st century will have very different challenges, many of them difficult to anticipate. But it is manifestly clear that changes are needed in legislation, protocols and practice within our public service to ensure that future generations understand the history of today."

Dr John Bowman is a historian and broadcaster who analyses Anglo-Irish affairs for The Irish Times’ coverage of the annual release of State Papers. His most recent book is Window and Mirror – RTÉ Television: 1961-2011, published by the Collins Press.