The YES3 Exporter Technical Guide

Intended Audience

This guide is intended for individuals who will be writing programs to process files generated by the YES3 Exporter. These include data managers and statistical programmers writing programs to import YES3 Exporter data into statistical systems such as SAS and R, as well as application developers seeking to integrate the YES3 Exporter into datamart generation and validation workflows.

The YES3 technical team

The following individuals participate in one aspect or another - usually multiple aspects - of YES3 software development.

Katy Araujo ninja, testing, documentation
Venugopal Bhatia, application developer
Peter Charpentier, lead EM developer
Kraig Eisenman, IT system engineer
Brian Funaro, lead IT system engineer
Mary Geda, program manager, documentation lead
Kaitlin Maciejewski, testing
Baylah Tessier-Sherman, ninja, testing
Janet Truebig, ninja, testing
Sui Tsang, lead ninja, testing lead, documentation

YES3 Exporter character encoding

The YES3 Exporter generates UTF-8 encoded text files. While not required by the UTF-8 specification, for compatibility the byte order mark (BOM) is included. We have found that the BOM is needed for Microsoft products like Excel. Rows are terminated by a new line (0A), but this might be implementation-specific (our hosts run Ubuntu 20.04 and PHP 7.4). It might be best to anticipate a possible CRLF (0D0A) row termination.

YES3 Exporter destinations

There are two export destinations: download and filesystem.

The download destination offers the options of downloading the exported data, the data dictionary, or a .zip package that includes both (see "The YES3 Exporter Payload," below).

To configure filesystem exports for a project, filesystem exports must be enabled by a REDCap administrator in the YES3 Exporter system EM configuration settings. Then a host filesystem target folder must be specified in the YES3 Exporter project EM configuration settings. The host filesyetem target folder should be a "bind mount" for a secure institutional fileshare.

Managing multiple datamarts

Since host filesystem mount points must be configured by ITS system administrators, and probably approved by an Information Security Officer, in most implementations filesystem exports will be reserved for large studies having the resources and a compelling case for automated datamart builds. However, filesystem export targets for large numbers of studies might be facilitated using the following approach:

1. Create a single mount point as a folder on a secure, very restricted institutional fileshare.

2. Create a subfolder for each study requiring a filesystem export. This will be that study's filesystem export target - but will not be accessible to that study.

3. Build an external application that will sweep through the daily exports, validate them against the REDCap database and if validated, pass each to a folder that is accessible to that export's study staff.

The YES3 Exporter Payload

When exporting to a host filesystem folder, or downloading to a .zip export package, a "payload" of three files is delivered:

The export name is "normalized" by converting to lower case and replacing spaces and other non-alphanumeric characters with underscores.

How export specifications are identified

The REDCap EM logging feature provides an elegant way to maintain specification backups. In order to track specification backups across EM log entries, each export specification is identified by the EM parameter export_uuid. This value is an RFC-compliant universally unique identifier, generated by the Javascript function YES3.uuidv4 function in js/yes3.js.

EM Log records

The YES3 Exporter makes extensive use of External Module logs, both for logging actions such as downloads and filesystem exports and for storing the export specifications and event prefixes. An EM log parameter named log_entry_typeindicates the function of the log record. Below is a listing of each log_entry_type value and the purpose of the associated EM log record.

The YES3 Exporter Information file

The YES3 Export Information file is a JSON-encoded object that provides information about the project and the export specification.

Object properties:

YES3 Exporter Variables

Export Data Layouts

The YES3 Exporter currently supports three export layouts: Vertical(v), Horizontal(h) and Repeating forms(r).

For all export layouts the first field or 'variable' in the exported record will be the recordid, and the variable name will be the recordid field name as defined for the specific project.

Depending on the layout and study design, the recordid variable may be followed by a collection of "REDCap Attribute" variables that indicate the REDCap data access group and event.

The remainder of the exported record is the collection of values for the selected fields, in the order specified. The variable name of each will be either the REDCap field name (vertical and repeating form layouts), or a composite of the REDCap event prefix and the REDCap field name (horizontal layout).

Following are more detailed descriptions of the export layouts.

The YES3 Exporter Data Dictionary

The YES3 Exporter Data Dictionary includes detailed metadata for every exported variable, as well as data distribution summaries.

One intended use for the YES3 Exporter data dictionary is for external applications to generate and execute fine-tuned data import code. For example, the observed maximum string length can be used to tailor SAS variable attributes to exactly match the exported data. Once a datamart has been rebuilt, an external process can use the data distribution information in the data dictionary to validate the import - or to create and deploy a simple summary data report.