Database preservation

Database preservation usually involves converting the information stored in a database to a form likely to be accessible in the long term as technology changes, without losing the initial characteristics (context, content, structure, appearance and behaviour) of the data.^[1]

With the prevalence of databases, different methods have been developed to aid in the preservation of databases and their contents. These methods vary depending on database characteristics and preservation needs.^[2]

There are three basic methods of database preservation: migration, XML, and emulation.^[1] There are also certain tools, software, and projects which have been created to aid in the preservation of databases including SIARD, the Digital Preservation Toolkit, CHRONOS, and RODA.

The characteristics of the database itself are taken into consideration when attempting preservation of said database. Relational databases are made up of tables which contain data in records and these tables then connect to one another through common data points that are stored in their records.^[3] However, with the emergence of big data the new NoSQL database is also coming into play.^[4] Databases are characterized as open or closed and static or dynamic. When a database is considered to be open it means it is open to additional data being added, however when a database is considered to be closed it means the opposite—that it is closed to new data because of its completed nature. A database is considered to be static when it contains records that are not edited or changed after their initial inclusion, however a database is considered to be dynamic when it contains records that may be edited in the future. Whether a database is open and static, open and dynamic, closed and static, or closed and dynamic will affect the methods used for preservation. It is more difficult to preserve a dynamic database than a static one because the data is constantly changing, and it is more difficult to preserve an open database than a closed one because data is constantly being added. The more often a database changes, either within a record or by adding a record, the more often steps must be taken to capture that change for preservation.^[2]

Three core methods of digital preservation can be applied to the preservation of databases as well. These methods include migration, XML, and emulation.^[1]

The migration method (also known as inactive archiving)^[3] involves transferring data from an obsolete database program to a newer format. There are three methods of migration: backward compatibility, interoperability, and conversion to standards. Backward compatibility involves utilizing newer software or hardware versions to open, access, and read a document which was made using an older version. Interoperability involves decreasing the possibility of obsolescence by ensuring a particular file can be accessed with more than one combination of software and hardware. Conversion to standards involves transferring data storage from a proprietary format to an open, more readily accessible, and widely used format.^[1]

The XML method (also known as XML normalization)^[3] involves converting original database information to the XML standard format. XML as a format does not require a particular hardware or software (beyond a text editor or word processor) and is both human and machine readable, making it a sustainable format for preservation and storage purposes.^[1] However, in converting data to XML format, certain interactive functionality of the database, such as the ability to query, is lost.^[3]

The emulation method involves recreating an older computing environment with newer technologies and software. This allows obsolete software, hardware, or file formats to remain accessible on new systems. Therefore, an outdated database could be run on an emulator which mimics the environment that the database was originally created in.^[1]

Version 1.0 of the Software Independent Archiving of Relational Databases (SIARD) format was developed by the Swiss Federal Archives in 2007. It was designed for archiving relational databases in a vendor-neutral form. A SIARD archive is a ZIP-based package of files based on XML and SQL:1999. A SIARD file incorporates both the database content and also machine-processable structural metadata that records the structure of database tables and their relationships. The ZIP file contains an XML file describing the database structure (metadata.xml) as well as a collection of XML files, one per table, capturing the table content. The SIARD archive may also contain text files and binary files representing database large objects (BLOBs and CLOBs). SIARD permits direct access to individual tables by exploring with ZIP tools. A SIARD archive is not an operational database but supports re-integration of the archived database into another relational database management system (RDBMS) that supports SQL:1999. In addition, SIARD supports the addition of descriptive and contextual metadata that is not recorded in the database itself and the embedding of documentation files in the archive.^[5] SIARD Version 1.0 was homologized as standard eCH-0165 in 2013.^[6]

Version 2.0 of the SIARD preservation format was designed and developed by the Swiss Federal Archives under the auspices of the E-ARK project.^[7] Version 2.0 is based on version 1.0 and defines a format that is backwards-compatible with version 1.0. New features in version 2.0 include:

• An upgrade of SQL:1999 support to SQL:2008 support
• Support for all SQL:2008 types, in particular user-defined data types (UDTs)
• More explicit validation rules for data type definitions using regular expressions
• Support for storing large objects outside of the SIARD file using “file:”URIs
• Support for “deflate” as a compression mechanism.

A XML schema was created by researcher José Carlos Ramalho from the University of Minho to capture table information and data from a relational database. It was published in 2007.^[8]

CHRONOS is a software product which serves as a database preservation tool.^[4] CSP Chronos Archiving represents one proprietary solution for database preservation. CHRONOS was developed from 2004 to 2006 by CSP in partnership with the University of Applied Sciences Landshut's department of computer science.^[4][9] CHRONOS pulls data from a database management system and stores it in a CHRONOS archive as text or XML files. All data can therefore be accessed and read without a Database Management System (DBMS), or CHRONOS itself, as it is in plain text format. This eliminates the need for maintaining a DBMS solely for reading preserved static databases as well as the need to, potentially riskily, migrate database files to newer database formats.^[9] Although CHRONOS stores data in plain text format, its querying capabilities, are considered comparable to that of a relational database.^[4]

A series of steps, created by the RODA project to ingest and preserve relational databases in a normalized format, represent the Database Preservation Toolkit or dbtoolkit: an instrument designed for the preservation and access of archived databases. Using the Database Preservation Toolkit, to achieve normalization of relational databases, data is converted to DBML (Database Markup Language) or SIARD, as both utilize XML, a standard format which does not require specific or proprietary software or hardware—ideal for a preservation format.^[10]

The Database Preservation Toolkit (DBPTK) allows conversion between database formats, including connection to live systems, for purposes of digitally preserving databases. The toolkit allows conversion of live or backed-up databases into preservation formats such as SIARD, an XML-based format created for the purpose of database preservation. In this conversion process the toolkit extracts unique DBMS information using DBMS-specific connectors. These connectors pair with a particular DBMS, extract its data, and represent it in XML form which then leads to representation in DBML and SIARD. New connectors can also be created for the ingestion of new DBMS’.^[10] The toolkit also allows conversion of the preservation formats back into live systems to allow the full functionality of databases. For example, it supports a specialized export into MySQL, optimized for PhpMyAdmin, so the database can be fully experimented using a web interface.

This toolkit was originally part of the RODA project^[11] and then released on its own. It has been further developed in the E-ARK project together with a new version of the SIARD preservation format.

The toolkit uses input and output modules. Each module supports read and/or write to a particular database format or live system. New modules can easily be added by implementation of a new interface and adding new drivers.^[12]

Research projects this regard include:

• Software independent archival of relational databases (SIARD)^[13]
• Software Database Preservation Toolkit (open-source, supports SIARD 2.0)^[12]
• Repository of Authentic Digital Objects (RODA)^[14]
• Digital Preservation Testbed^[15]
• Lots of Copies Keep Stuff Safe (LOCKSS) project was led by libraries at Stanford University.^[16]

RODA, or the Repository of Authentic Digital Objects, was a project launched in Portugal in 2006 by the Portuguese National Archives, in order to preserve those digital objects produced by Portugal’s government institutions. The project aimed to combine several types of digital objects into one repository including relational databases. As a singular repository of many differing types of digital objects, RODA aims to normalize all ingested objects, that is to minimize the format types utilized to store documents and to preserve like documents in like formats.^[10]

The RODA project emphasized the creation of a standardized method for preserving databases as digital objects. Database preservation poses a unique challenge in that the preservation process is split into three layers: data, structure (logic), and semantics (interface).^[17] That is, it was determined that the databases’ data, as well as its structure and semantics, need to be preserved. In order to preserve all three of these elements, the RODA project developed the Database Preservation Toolkit.^[10]

• Digital preservation