Understanding Lytics / Product Documentation / Data Onboarding and Management

Data Streams

All data sent to Lytics must be sent through a data stream. Data streams are silos of raw data containing key-value pairs organized by source that can be used in the mapping of user fields. Until the raw event of a data stream is mapped to a user field it will not be available for use to build audiences. This gives Lytics the ability to filter, aggregate, and merge raw data into user fields in a non destructive way.

NOTE: The number of keys in a data stream can be extremely large. This is normal and does not impact performance. Furthermore, any keys comprising less than 0.1% of the data volume will be hidden from display to reduce clutter.

Viewing your data streams

data streams section You can view information about your data streams in your Lytics dashboard by navigating to Data > Data Streams. The primary purpose of this section is to verify that data is successfully being received by Lytics. If your account has multiple data streams you can use the dropdown menu located above the graph to view a different stream.

Many integrations have multiple streams. For instance, it is common for email integrations to have an activity stream as well as a user stream. Integrations streams should be prefixed to help identify the source. You can find the streams for an integration under the documentation for that integration.

Data ingress graph

data streams data ingress graph The event ingress graph shows the number of events collected on a stream for the selected time period (past day, week, month, 3 months, and year) and interval (hourly, daily, weekly, and monthly). Above the graph you will find the time the last message from this stream was received, the source of the data stream, and the number of fields in the stream.

NOTE: Last message received strives for real time reporting but can lag under a number of conditions including during bulk imports. If a data stream is not updating as expected please contact support for assistance.

Raw keys table

Below the event ingress graph is the raw keys table. An event may contain any number fields containing a key-value pair. Each record in this table is a unique raw key seen on the stream.

data streams raw keys table

The table has the following information on keys:

ColumnDescription
NameThe name of the key.
Predicted TypeThe assumed data type for the value, determined by sampling the values received.
First SeenThe date the key was first seen, according to the date on the event.
Last SeenThe last time an key was seen, according to the date on the event.
Times SeenThe number of events that contained the key.
Unique ValuesThe number of different values seen.
Times UsedThe number of user fields that use the key.

In addition to these seven columns, each record in the table can be clicked to open up a set of sample values. This can be used to verify that values are being collected and they match the expected data.

NOTE: If a key has many different values, the modal may not display all the values for the chosen key.

The table can be filtered in three ways: used vs. unused, common vs. uncommon, and text search.

FilterDescription
UsedA raw key is that mapped to a user field.
UnusedRaw keys that are collected and stored, but never mapped to a user fields.
CommonRaw keys that have been seen more often on events relative to other raw keys based on the times seen value.
UncommonRaw keys that are seldom seen on events relative to other raw keys.

Hiding Keys

Raw event keys can be hidden but it is important to note that once hidden, keys cannot be made visible again through the user interface - only through the API. It is recommended that a list is kept of hidden keys in the event one needs to be resurfaced at a later date. If you need assistance, please contact us at [email protected] with the key name and your account ID.

To hide a key:

  1. Select the checkbox next to the name of the key or keys.
  2. Click Remove selected key.

Monitoring data streams

As mentioned in the previous sections, the Data Streams section is primarily used to monitor the collection of raw data.

The data ingress graph displays the time and volume of raw events for each stream. You can use it to evaluate when data has stopped being sent to Lytics entirely or when the volume of events changed.

The raw keys table can be used to check on individuals keys including when they were last seen, the type of data being recorded, and the sample values being recorded.