Google Cloud Pub/Sub: Import Events
Importing user and activity data from Cloud Pub/Sub in real-time results in new users or existing user profiles supplemented with activity data. You can use this data to build and refine your existing Lytics audiences to power better, cross-channel campaigns.
The integration is similar to AWS Kinesis Streaming or Azure Event Hubs. It requires you to set up a Cloud Pub/Sub Topic, which you publish JSON messages to. Lytics then listens (subscribes) to this Topic and receives those messages.
- Implementation Type: Server-side.
- Implementation Technique: REST API.
- Frequency: Real-time.
- Resulting data: Raw event data, and user fields, or content (if mapped).
This integration utilizes the Cloud Pub/Sub API methods of receiving events from a publisher. Specifically we use the Golang library. You are the publisher who sends (publishes) events through a user-created object known as a Topic. Lytics manages an object called a Subscription, which receives the data (events) from the Topic you created. The continuously running workflow works like this:
- Through the service account token you created in the authorization step, Lytics creates a new client, which manages and receives events using the subscription.
- Lytics verifies the Topic ID you provide in the workflow configuration, and creates a connection to the Subscription associated with this Topic. If no Subscription exists or is provided, Lytics creates a new Subscription in the following format:
- The Topic is the "publisher", whereas the Subscription is the "receiver". This means you must publish/send data to this Topic from your account, which Lytics will then receive and process.
- The workflow will continue to get events in real-time as soon as events are published on the Topic.
- The events are sent to a user-selected stream which then gets added to user profiles or creates new user profiles.
Data imported by this workflow will be sent to a new or existing data stream in your account which you can define as part of the configuration. You will need to map the raw event data into user or content fields using LQL. You may reach out to Lytics support for assistance with mapping your custom data.
Follow these steps to set up and configure an import of events in the Lytics platform.
- Navigate to Google Cloud in the Integrations section of Lytics.
- Select Workflows and then click New workflow.
- Select Cloud Pub/Sub Import.
- Select the Cloud Pub/Sub service account authorization you would like to use.
- In the Stream text box, either select or type in the stream you would like to send these events to.
- Enter the Project ID associated with the Cloud Pub/Sub topic Lytics will receive from.
- Enter the Topic ID you would like Lytics to receive events from.
- (Optional) Enter an already existing Subscription Name that you would like to receive events on or leave it blank and Lytics will create a new one named
- By default, after 100 bad messages (usually because they are not JSON), the workflow will error. If you would like to change the maximum number of bad messages before the workflow errors, enter a number under Max Bad Messages.
- Click Start Import to begin importing data from your Pub/Sub topic.