Custom Integrations: CSV Import
Lytics allows you to import data via a CSV file from data sources we do not natively integrate with.
- Implementation Type: Server-side.
- Implementation Technique: File based transfer.
- Frequency: Batch, frequency can be configured.
- Resulting data: User profiles once the imported data is mapped via LQL.
This integration parses CSV files and imports fields and associated data. Please see Custom Data Ingestion for more information on file naming, field formatting, headers, timestamps, etc. Once the integration is started it will:
- Connect to the SFTP server.
- Navigate to the configured folder.
- Read the selected CSV file.
- Import the fields that are chosen during configuration.
Once you choose the file to import during configuration, Lytics will read the file and list all the fields that can be imported. You may select fields that you wish to import.
Follow these steps to set up and configure an import job for Custom integration in the Lytics platform. If you are new to creating jobs in Lytics, see the Jobs Dashboard documentation for more information.
- Select Custom from the list of providers.
- Select the Import CSV job type from the list.
- Select the Authorization you would like to use or create a new one.
- Enter a Label to identify this job you are creating in Lytics.
- (Optional) Enter a Description for further context on your job.
- Complete the configuration steps for your job.
- In the Stream text box, select the data stream where the data will be posted or enter a name to create a new stream.
- (Optional) From the Folder input, select the folder where the CSV file is located.
- From the File input, select the file to import, or if this is a continuous import of dated files e.g.,
file_20160218.csv, choose the earliest dated file to import, and all files will be imported from that date forward.
- (Optional) In the Custom Delimiter text box, enter the delimiter of the file. Default delimiter is a comma,
,. Other common delimiters may be "|" (pipe) or ";". For tab delimited files enter "t".
- (Optional) From the Timestamps input, select one or more columns in the CSV that contain the timestamp of an event. Sort in order of importance (e.g., "Updated" should be above "Created"). If no fields are specified, the event will be time stamped with the time of the import. Timestamp selection is unavailable for zip files larger than 100MB.
- (Optional) From the Fields input, select fields to import. If left empty all fields will be imported. If no field names appear, check to ensure the CSV file has an appropriate header row or the delimiter may need to be changed. Field selection is unavailable for zip files larger than 100MB.
- (Optional) In the Prefix text box, enter the file name prefix. You may use regular expressions for pattern matching. The prefix must match the file name up to the timestamp. A precalculated prefix derived from the selected file will be available as a dropdown.
- (Optional) Select the Keep Updated checkbox to run this import continuously checking for new files periodically.
- (Optional) From the File Upload Frequency input, select how often to check for a new file.
- (Optional) From the Time of Day input, select the time of day to start import.
- (Optional) From the Timezone input, select the timezone for the time of day you selected above.
- (Optional) Select the Diff checkbox to compare file contents to the previous file contents during continuous import and import only rows that have changed. This is useful when large amounts of data remain unchanged in each file.
- Click Start Import.