Data Import

This section describes all of the possible options on the Import Data tab in Data Manager (shown below):

Data Import Data File Format

The data import data file must be a tab-delimited text file where the first line of the file contains variable names delimited by tabs that correspond to the survey variable names. Subsequent lines in the data file correspond to a survey participant entry or submission.

Tab-delimited text files created by Microsoft Excel are also supported, which includes triple-quoted data values. Triple-quoted data values are created by Excel whenever a data value in a cell contains one or more double quote characters. Whenever this type of data is included in a spreadsheet and the spreadsheet is saved as a tab-delimited text file, the values are written out with a leading and trailing double quote character and each double quote character in the original value is converted to two double quote characters.

For example, the phrase “Hello” will actually be written out as “””Hello”””.

Data Import Actions

The data import actions control how the data is imported or if it is at all.

• Test file for errors only (do not import data) – This is the default data import action. This action doesn’t actually import the data but rather checks the data for any errors. This action is the recommended first step when importing data. Any errors reported will need to be investigated and corrected before the data import will actually work.
• Import as test data – This action imports the data as test data and allows users to run queries against the data which can ensure that Data Import process worked as expected. The benefit of this action is that test data can be quickly removed and the data import can be re-tried again if necessary.
• Import as production data – This action will import the data as production data. This option should be selected after you have ensured the results of importing the data are correct using the other two actions. Once data is imported as production data it is difficult and time-consuming to delete it.
• Update existing submissions – This action allows for the addition or correction of existing survey submissions.  This action must have the DatStat.SESSIONID as one of the columns in the upload file.

Data Cleanse Option

This option may be desirable if the survey contains questions or collections with Show-If. When this option is enabled responses will be discarded if Show-If would have caused this question or parent collection to not be shown.

Error Reporting Options

Data Import will only import data if there are no reported errors. These options allow for users to quickly suppress errors of a certain category especially if the error conditions are determined to be intentional or benign.

• Suppress unknown column errors – Selecting this option will prevent errors of type Unknown Column Error from being reported. This error condition is reported when there are column names in the first row of the data file that do not match any variable names in the survey. A data file might contain additional columns that are not in the destination survey. Checking this box will essentially ignore such columns.
• Suppress missing column errors – Selecting this option will prevent errors of type Missing Column Error from being reported. This error condition is reported when there are survey variables in the destination survey that are missing from the data file. A data file might not include all variables that are listed in the destination survey. Checking this box will ignore this condition as an error condition and will allow the data import to proceed.
• Suppress unknown scale value errors – Selecting this option will prevent errors of type Unknown Scale Value Error from being reported. This error condition is reported when there is a specific data value that is not defined in the scale of the corresponding survey variable. When this error is suppressed, the data is cleared and treated as unanswered for these types of conditions.
• Suppress response guide validation errors – Selecting this option will prevent errors of type Response Guide Validation Error from being reported. Errors of this type are reported when the values in the data file violate the following response guides specified for the survey question:
• 1. Minimum/maximum length
  2. Format (meta-type)
  3. Lower/upper bounds
  4. Data type violation (e.g. non-numeric data specified when numeric data type expected). When this error is suppressed, the data is cleared and treated as unanswered for this question.

The Data Import is performed using the Data Manager. Below are some simple steps designed to help guide the Data Import process.

1. In the Data Manager.
2. Click on the desired survey to import data into
3. Select the Import Data tab *
4. Provide the name of the tab-delimited data file
5. Click the Upload button and check for any errors
6. If there are errors, either correct them in the data file or chose to suppress them by selecting the Error Reporting options on the Import Data tab
7. (Optional step) After no errors are reported, select the data file again, click the Import as Test Data option, and then click the Import button. After the data has been imported as test data, run some queries and make sure the data was imported properly. This test data can be quickly removed by selecting the Test Data tab and clicking the Delete Rows button
8. To import into production select the data file again, click the Import as Production Data option, and then click the Import button

NOTE: that undoing this operation is a very time intensive process and requires a user to manually delete each submission. It is recommended to import the data as test data first as outlined in the previous step.

NOTE: Survey data for Data Import must be specified as a tab-delimited text file with the variable names as the first row of the file.

NOTE: Data Import must be enabled for each user that will be using the feature.

* If the Import Data tab is not visible see the system administrator. This might be due to the fact that the Data Import feature has not been purchased, or Data Import hasn’t been enabled in the Illume license, or Data Import hasn’t been enabled for the user currently logged in.

DatStat Internal variables set by the Data Import process

Values for the following list of DatStat Internal variables are set by both the Data Import and Remote Data Collection components.

DATSTAT.UPLOADDATETIME

Values for this variable indicate the date and time this survey session was imported. This variable is also set when data has been uploaded using Remote Data Collection.

DATSTAT.UPLOADTYPE

Possible values for this variable are “1 – Remote Data Collection” and “2 – Imported”. Data Import will set this variable to a value of 2.

DATSTAT.UPLOADUSER

This variable will be set to the user name of the user that performed the data import for each survey session. This variable is also set when data has been uploaded using Remote Data Collection.

Importing data into Internal DatStat variables

It is possible to import data into the most of the internal DatStat survey variables. However, the following list of internal variables is not supported. There is no harm in including these variables in the data file, but they will be ignored without any errors being reported.

• DATSTAT.SUBMISSIONID
• DATSTAT.ELAPSEDTIME
• DATSTAT.PCTCOMPLETE
• DATSTAT.UPLOADDATETIME
• DATSTAT.UPLOADUSER
• DATSTAT.UPLOADTYPE
• DATSTAT.QUESTIONSUNANSWERED
• DATSTAT.PCTUNANSWERED

Importing data into the DATSTAT.SESSIONID variable

The final value for this variable in the imported survey session will ALWAYS be different than the value specified in the data file. The purpose of specifying this column is so the same participant can be stamped in the newly created imported survey session as the previous session identified by this value.

In general it is a good idea to include this column in the data import data file if the data being imported was downloaded from a survey in the SAME DatStat Illume system. This column is ignored and a new value will be provided if importing data that originated from a different DatStat Illume system or a survey system other than DatStat Illume.

This participant value isn’t exposed via the Data Manager, but it is used to retrieve the survey session of a participant when a participant is resuming a survey or when the system needs to determine if the participant already took the survey. This information is also used when determining whether to send an email to a participant based on the criteria set in the email job.