Understanding Lytics / Product Documentation / Data Management

Using Data Streams

The Data Streams section of Lytics can be found under Data at Data > Data Streams.

Data streams as seen in the product

What are Data Streams?

Data streams can be thought of as containers for different data. When data gets sent to Lytics, it is also always additionally sent to a data stream. The default data stream, is aptly named default. Keeping data organized in this manner allows for clean filtering, aggregation, and most importantly, merging.

The most important detail to consider about data streams when it comes to data management is that they represent raw data. The graph visualizes the volume of data of this raw data and lists the keys it contains. Remember that audience membership is based on user fields which are essentially keys that have been mapped for use. This means that seeing data in data streams does not mean the data is available for use in audiences. It must first be mapped into user fields before it can be used.

Note: the number of keys in a data stream can be extremely large. This is normal and does not impact performance. Furthermore, any keys comprising less than 0.1% of the data volume will be hidden from display to reduce clutter.

Read more about User Fields in terms of data management.

Viewing Real-time Processing

The primary purpose of the data streams tool is to verify that data is in fact being collected by Lytics. The easiest way to confirm this is by reading the event ingress graph.

The event ingress graph

The graph shows the number of events collected on a stream in bi-hourly intervals. Standard intuition applies here: when there are no bars, no data is being collected.

Above the graph are a few helpful facts.

  1. Last Message Received: This is the timestamp on the last event that was collected.
  2. Data Source: The integration the stream is associated with.
  3. Fields: The number of different field keys seen on the stream.

Fields are important to understand. They represent the smallest workable unit of data in Lytics. The table below the graph is designed to help discover and validate individual fields.

Finding Raw Event Details

Underneath the event ingress graph is the raw events table. Each record in this table is a unique raw field seen on the stream. All events are packets of keys and values. A raw field represents the key.

The raw events table

The table has the following information on fields:

  1. Name: The name of the fields, which is equivalent to the key that was sent in an event.
  2. Predicted Type: The assumed data type for the field. Determined by sampling the values received.
  3. First Seen: The date the event was first seen, according to the date on the event.
  4. Last Seen: The last time an event was seen, according to the date on the event.
  5. Times Seen: The number of events that contained the field.
  6. Unique Values: The number of different values seen for the field.
  7. Times Used: The number of User Fields that are a view of the raw field.

In addition to these seven columns, each record in the table can be clicked to open up a set of sample values seen for the raw field.

The sample values modal

The table can be filtered in three ways: used vs. unused, common vs. uncommon, and text search.

  • Used: A raw field is used when it is a component of a User Field. Equivalent to all raw fields with Times Seen greater than zero.
  • Unused: The opposite of Used. These raw fields are collected and stored, but never used in User Fields, and thus never seen in User Profiles.
  • Common: Common raw fields have been seen very often on events relative to other raw fields. This is based on the Times Seen value.
  • Uncommon: The opposite of Common. These raw fields are seldom seen on events relative to other raw fields.

Monitoring Streams And Integrations

An active Lytics account that is collecting data from a variety of sources will have multiple data streams continuously collecting data. Use the dropdown on the top right of the Data Streams page to switch streams and verify that data for the stream is being collected as expected.

A sample streams dropdown opened

Note: Many integrations have multiple streams. For instance, it is common for email integrations to have an activity stream as well as a user stream. Learn the integration prefixes to identify all the streams for an integration. Also keep in mind that the Data Source mentioned above the ingress graph will be the name of the integration for the stream.