Google BigQuery: Export Activity Data

Export event data from any Lytics data stream to Google BigQuery. NOTE: Unlike user fields, events are not represented within the Lytics dashboard. Events are the raw data received from integrations as seen in your data streams.

• Implementation Type: Server-side.
• Implementation Technique: REST API.
• Frequency: Batch.
• Resulting data: Raw event data is stored in the resulting table.

This integration utilizes the Google BigQuery APIs to send user data. Once the export is started the job will:

1. Check if dataset exists, if not create it in Google BigQuery.
2. Check if a table exists, if not create one.
3. Generate a CSV for each table to be created or updated in BigQuery by:
   1. Scan through events on the data stream.
   2. Generate CSV row(s) for each scanned event.
   3. Write CSV row to Google Cloud Storage.
4. Upload the CSV(s) to BigQuery.

There are a few things to know when running an event export:

• The event export scans the complete data stream from the first collected event to the current event, unless otherwise specified by the export configuration.
• The export will run continuously throughout the day.
• Field names in BigQuery will not be an exact match of field names in Lytics. Names will be formatted to fit BigQuery's field naming schema.
• If two field names are transformed to BigQuery format and conflict, one will get an integer appended to the end of the field name. For example, if both first_name and First Name exist in Lytics and are exported, they'll be written to first_name and first_name1 in BigQuery.
• Low usage fields (fields populated on <0.1% of total volume for the exported stream) will not be included within the event export.

The fields included will depend on the raw event in Lytics data. All fields in the selected stream will be included in the exported table. To see a list of fields in a stream, select the stream name from Data > Data Streams in the Lytics platform.

Follow these steps to set up and configure an export of events from Lytics to Google BigQuery. If you are new to creating jobs in Lytics, see the Jobs Dashboard documentation for more information.

1. Select Google Cloud from the list of providers.
2. Select the Export Activity Data job type from the list.
3. Select the Authorization you would like to use or create a new one.
4. Enter a Label to identify this job you are creating in Lytics.
5. (Optional) Enter a Description for further context on your job.
6. From the BigQuery Project input, select the Google BigQuery Project you want to export data to.
7. From the BigQuery Dataset input, select the Google BigQuery Dataset you want to export data to.
8. From the Data Streams to Export input, select data streams to export. A stream is a single source/type of data. You may choose more than one. If none are selected, all streams will be exported.
9. In the Maximum numeric field, enter the number of events to be exported. If left blank, all events will be exported.
10. In the Start Date text input, enter the date of the oldest event you want to export. Events from this date onwards will be exported. Use yyyy-mm-dd format (e.g. 2015-08-13).
11. In the End Date text input, enter the most recent date you want to export. Events before, but NOT including this date will be exported. Use yyyy-mm-dd format (e.g. 2015-08-13).
12. Click on the Show Advanced Options tab to expand the advanced configuration.
13. Select the Partitioned Table checkbox to use partitioned tables that can help lower query costs by allowing you to target a specific date range of data instead of the whole data set. Tables will be partitioned by the event date.
14. From the Partition Duration dropdown, select the duration of the table's partitions. This is ignored if Partitioned Table is not selected.
15. Select the Flatten Streams checkbox to export all streams to one BigQuery table.
16. Select the Keep Updated checkbox to continuously run this export.
17. Select the Start Export From Now Onwards checkbox to only export events collected from now onwards. This will override any start or end date configuration made.
18. Click Start Export.