Google Cloud Storage: Import Files

Import user activity and profile files from Google Cloud Storage into Lytics.

• Implementation Type: Server-side.
• Implementation Technique: File based transfer.
• Frequency: Batch.
• Resulting data: Raw event data.

Once initiated, the import job will:

1. Retrieve a list of objects in the bucket that match the prefix.
2. Sort the files by date, and ignore any files older than the last file.
3. Iterate through the files.
4. Convert each row/JSON object into an event.

Due to the nature of the files that can be imported by the File import, no default LQL is available. All fields available will be added to the data stream.

Follow these steps to set up and configure an import job for Google Cloud in the Lytics platform. If you are new to creating jobs in Lytics, see the Jobs Dashboard documentation for more information.

1. Select Google Cloud from the list of providers.
2. Select the Import Files from the list.
3. Select the Authorization you would like to use or create a new one.
4. Enter a Label to identify this job you are creating in Lytics.
5. (Optional) Enter a Description for further context on your job.

6. Complete the configuration steps for your job.

   

7. In the Project text box, enter the Google Project ID you want to import data from.

8. In the Bucket text box, enter the Google Cloud Storage bucket you want to import data from.
9. (Optional) In the Starting File text box, select the file you would like to import first. Only newer files will be imported.
10. (Optional) In the Prefix text box, enter the file name prefix to filter files by. Example: new-users-
11. (Optional) In the Since text box, enter the oldest date to import files since. Only files newer than this date will be imported. Use yyyy/mm/dd HH:MM/SS format. If left blank all files created after Starting File will be imported.
12. From the File Type input, select the type of file. Select CSV if data is formatted as a CSV with header row (CSV without a header row is unsupported). Select JSON if the file is formatted as a JSON lines file.
13. (Optional) In the Timestamp Field text box, enter the field containing the event timestamp.
14. (optional) Select the GZipped checkbox if the file is gzip compressed.
15. (Optional) Select the PGP Encrypted checkbox if the file is PGP encrypted.
16. (Optional) In the Stream text box, enter the data stream to add events to. If left blank, events will be added to the gcs_files data stream.
17. (Optional) Select the Continuous Import checkbox to import continuously.
18. (Optional) From the File Import Frequency input, select how often the import should look for new files.
19. (Optional) From the Time of Day input, select time of day to start the import each day.
20. (Optional) From the Timezone input, select timezone for time of day.
21. Click Start Import.