Digital Archives Transfer and Processing Workflow

This guide is part of the RBSC Archives Toolkit.

Created by Jacky Lai, Digital Archivist.

1. Transfer of Digital Records to University Archives

The transfer process establishes a secure, documented copy of the donor's digital records in University Archives (UA) custody. The method used depends on whether UA staff have direct access to the donor's computer or whether the donor provides a physical storage carrier.

Transfer Using DART

Use DART when UA staff have access to the donor's computer and can perform the transfer directly from the donor's system.

DART can be used to package the records into BagIt bags before they leave the donor's computer. Creating the bag on the donor's computer provides a structured transfer package and allows the contents to be documented and validated as part of the transfer process.

Identify the digital records being transferred and confirm the scope of the transfer with the donor.
Connect to or access the donor's computer using the agreed-upon transfer method.
Use DART to create a bag containing the records.
Confirm that the bag has been successfully created and that the contents correspond to the records being transferred.
Transfer the completed bag(s) from the donor's computer to UA-controlled storage using TeraCopy.
Transfer the bag(s) to the appropriate UA laptop, USB storage device, SSD, or server.
Retain the original donor files on the donor's computer until the transfer has been successfully completed and UA has confirmed receipt, unless the donor has specifically requested otherwise.

Transfer from a Donor-Supplied Carrier

Use TeraCopy when the donor provides records on a physical carrier, such as a USB flash drive, external hard drive, optical disc, or other removable media.

Before copying the records, document the carrier and establish an initial inventory of its contents.

Photograph the carrier and accompanying labels
Connect the donor-supplied carrier to the UA workstation/laptop using appropriate precautions.
Do not modify, reorganize, or delete files on the donor's carrier.
Run DROID against the records on the donor-supplied carrier.
Review the results of the initial identification, including file formats, extensions, and other relevant information.
Use TeraCopy to copy the records to UA-controlled storage.
Transfer the records to the appropriate UA laptop, USB storage device, SSD, or server.
Use TeraCopy's verification functionality to confirm that the copied files match the source files.
Retain the donor-supplied carrier according to UA's established custody and disposition procedures [TBD].

The donor's original carrier should be treated as the source copy during this stage. Processing should be performed on a UA working copy rather than directly on the donor's original files.

2. Establish a Working Copy

Once the records are in UA custody, create a working copy before beginning digital forensic analysis or processing.

The purpose of the working copy is to provide a version of the records that can be examined, analyzed, arranged, and otherwise processed without altering the original transfer.

Confirm that the transfer from the donor was completed successfully.
Use TeraCopy to create a working copy of the transferred records.
Store the original transfer and working copy separately.
Clearly identify which copy is the original/source copy and which is the working copy. [Naming convention TBD]
Create an “Archivist notes” folder for storing processing documentation and generated reports.
Record the location of both copies in processing documentation.
If the records are subsequently moved to another storage location, use a verified copy process and document the movement in processing notes.

The original/source copy should remain untouched throughout processing. Any changes made during appraisal, arrangement, normalization, or other processing must occur on the working copy or other copies derived from the working copy.

3. Digital Forensics and Initial Analysis

Digital forensic analysis is used to establish a detailed understanding of the records before appraisal and arrangement. The initial analysis should be performed on both the original/source copy and the working copy.

The analysis provides information about the volume, file formats, structure, potential duplicates, potentially sensitive information, and other characteristics of the records.

DROID

Run DROID on both the original/source copy and the working copy.

DROID is used to identify file formats based on file signatures and to provide an initial inventory of the digital objects.

Record relevant information such as:

File names and extensions
Identified file formats
File format versions, where available
Number of files
Number of unidentified files
Locations of files within the directory structure
Other format-identification information relevant to processing
Checksums

Compare the results from the original/source copy and working copy. Unexpected differences should be investigated before proceeding.

TreeSize

Run TreeSize on both the original/source copy and the working copy.

TreeSize provides information about the size and structure of the records and can be used to identify:

Total size of the transfer
Number of files and folders
Distribution of files across folders
Large files
Empty folders
Deep directory structures
Potential duplicate files

Use TreeSize's file and folder information to identify potential duplicates. Pay particular attention to files with identical names, sizes, locations, or other characteristics that suggest they may be duplicate copies.

TreeSize's duplicate-identification results should be treated as an initial screening tool. Where necessary, confirm duplicates using checksums before removing or excluding material.

Preserve the TreeSize output as part of the processing documentation.

Click here for a more detailed guide on how to use TreeSize in the context of processing born-digital records.

Brunnhilde and Bulk Extractor

Run Brunnhilde on the Linux workstation, including its Bulk Extractor component, on both the original/source copy and the working copy.

The results can provide additional information about the contents of the records, including file formats, potential personally identifiable information (PII), and other features that may not be apparent from file names and extensions alone.

Review the generated reports for issues requiring further investigation.

Attention should be paid to:

Potential PII
Email addresses
Names and other identifying information
Potentially sensitive documents
Unusual or unidentified files
Duplicate or near-duplicate material
Files that may require additional format identification or examination
Other anomalies identified by the forensic tools

4. Review, Appraisal, and Arrangement

The information generated during digital forensic analysis is used to support appraisal and arrangement decisions. Consult with the supervising archivist before proceeding with appraisal and arrangement.

Identify Duplicates

Use the results from TreeSize, DROID, Brunnhilde to identify potential duplicates.

Duplicates should be reviewed before deletion or exclusion from the archival transfer. Files that appear to be duplicates based only on file names, locations, or file sizes should not automatically be treated as identical. Where necessary, use checksums to confirm whether files are identical. Document significant duplicate-removal decisions in processing documentation.

Identify PII and Sensitive Information

Review the results of Brunnhilde’s Bulk Extractor and manual review of files to identify potential PII or sensitive information.

Examples include:

Email addresses of third party
Contact information
Account information
Financial information
Potentially sensitive personal or institutional information

The identification of potential PII is an initial screening activity. A positive result does not necessarily mean that the material must be restricted or removed. Potentially sensitive records should be reviewed in accordance with UA's applicable privacy, access, and records management requirements.

Document significant findings and any resulting access or processing decisions in processing documentation.

Appraisal

Appraisal may involve:

Identifying records that fall outside the scope of the acquisition
Identifying duplicate material
Identifying transitory or non-record material
Identifying system-generated or administrative files that do not have long-term archival value
Identifying records that require additional review

While it is not necessary to document every appraisal decision at the digital file level, it is good practice to record the criteria and major decisions made during appraisal in processing documentation.

Arrangement

Arrange the digital records according to the proposed archival arrangement. For more information about this process, consult the Born-Digital Records: Arrangement, Description, Identifiers Wiki.

Where the original directory structure has archival value, retain it where practical. Where the original structure is unsuitable for archival description or access, records may be reorganized according to the approved archival arrangement.

Changes to the original structure should be documented in processing documentation.

5. Reanalysis After Processing

Digital forensic analysis should be repeated when significant changes are made to the working copy.

For example, rerun DROID, TreeSize, and/or Brunnhilde after:

Removing duplicates
Removing material during appraisal
Reorganizing files
Flattening or restructuring directories
Adding or removing files, including adding normalized files
Performing other significant processing activities

The purpose of reanalysis is to ensure that the descriptive and technical information being recorded reflects the final processed records, rather than the original transfer.

Reports generated before and after processing may be retained when they provide useful documentation of the processing history. Be sure to assign file names to these reports to reflect whether they are the generated for the final processed records or a previous working copy, or for the original transfer. [Naming convention TBD]

6. Archival Description

Once appraisal and arrangement are sufficiently complete, create the archival descriptions for the processed records. For more information about this process, consult the Born-Digital Records: Arrangement, Description, Identifiers Wiki.

Authority Record

Create or update the appropriate authority record for the creator of the records.

The authority record should establish the authorized form of the creator's name and provide relevant biographical or administrative information as required by UA's descriptive practices.

Where an existing authority record is available, review it for accuracy and update it as necessary.

Fonds-Level Description

Create or update the fonds-level description to provide an overview of the entire archival unit.

The description should provide appropriate information about:

Creator
Administrative/biographical history
Custodial history
Scope and content
Date range
Extent
Arrangement
Access and use conditions
Related materials
Other relevant descriptive information (record in the notes area)
Subject Access Points (Assign up to 5 SAPs, check with supervising archivist if you think more SAPs should be assigned)

Series-Level Descriptions

Create or update the series-level description(s) as needed.

File- and Item-Level Descriptions

Create file- and/or item-level descriptions according to the guidelines in the:

7. Generate Technical Documentation

Generate a final TreeSize HTML report for the processed records.

The report should represent the records after appraisal and arrangement and should therefore correspond as closely as possible to the material being preserved.

The report can provide researchers and archivists with a browsable representation of the digital records' directory and file structure.

8. Enter Descriptions into AtoM

Enter the completed archival descriptions into AtoM.

The hierarchical structure in AtoM should reflect the approved archival arrangement.

As applicable, create or update:

Authority record
Fonds-level description
Series-level descriptions
File-level descriptions (typically via CSV import)
Item-level descriptions (typically via CSV import)

Where a TreeSize HTML report has been generated, add a link to the report in the appropriate AtoM description. The link should allow researchers or staff to navigate to the technical representation of the digital records where access to the report is appropriate.

Before finalizing the description, verify:

Dates and extent are accurate
Description hierarchy is correct
Scope and content accurately reflects the records
Physical/digital extent is accurate
Access restrictions are correctly represented
Digital objects or technical documentation are linked appropriately
The TreeSize report link works
Authority records and access points are correctly associated

9. Prepare and Ingest SIPs into Archivematica

After appraisal, arrangement, description, and final validation are complete, prepare the records for preservation ingest into Archivematica.

The final preservation package should represent the records that UA has determined should be retained as part of the archival collection.

Before ingest:

Confirm that appraisal and arrangement are complete.
Confirm that the final file set corresponds with the archival descriptions.
Confirm that unnecessary duplicates and excluded material have been removed from the preservation set.
Confirm that the records have undergone the required format identification and technical analysis.
Confirm that the final working files are complete and accessible.
Prepare the appropriate SIP(s) according to UA's established packaging conventions [TBD].
Transfer the SIP(s) to Archivematica.
Begin the Archivematica ingest and preservation workflow.
Review ingest results and investigate any errors or normalization failures.
Confirm that the resulting AIP(s) have been successfully created and stored.
Record the relevant Archivematica identifiers and ingest information in the processing documentation in the “Archivists notes” folder.

Any normalization failures or other preservation-processing errors should be reviewed before the workflow is considered complete. Where a normalization failure is expected or unavoidable, document the failure and the decision about how the original file will be preserved and/or made available.

10. Final Quality Control and Completion

The workflow is complete when the records have been successfully transferred, processed, described, and preserved, and the relationship between the archival description and preservation system has been documented.

The final review should confirm that:

The original/source transfer remains intact.
A working copy was created and used for processing.
Digital forensic analysis was completed.
Significant appraisal and arrangement decisions were documented.
Final DROID, TreeSize, and Brunnhilde analysis were completed where required.
Final archival descriptions accurately represent the processed records.
A TreeSize HTML report has been generated.
AtoM descriptions have been entered and reviewed.
The TreeSize report has been linked from AtoM where appropriate.
SIPs have been successfully ingested into Archivematica.
Preservation processing errors have been resolved or documented.
Locations of the original transfer, working files, preservation packages, and access materials are documented.
Relevant checksums, reports, processing notes, and other technical documentation have been retained according to UA's digital preservation procedures.

The final documentation should provide a clear record of what was received, what was done to the records, what was retained or removed, how the records were described, and how they were transferred into preservation storage.