Interpreting Model Messages

Building effective Lookalike Models is an iterative process. To help with these iterations, Lytics provides diagnostic messages that identify ways to improve your model performance.

In general, these model messages can help you both select the best source and target audiences for your model, and create the best Predictive Audience from your model.

Most of the time, Lookalike Models are built to drive users toward conversion. This flow should be reflected in how you consider selecting the right source and target audiences to build your model.

• Source Audience: the group of users to reach with your marketing messages, such as “unknown users.”
• Target Audience: the group of users that represents the desired outcome for users in the source audience, such as “users with email addresses.”

For more definitions of Lookalike Model terminology, see the Core Concepts section.

When defining your source and target audiences, try selecting audiences that are at adjacent stages in your customer lifecycle, rather than at divergent ends of your funnel (like Brand New Visitors and Multi-purchase Premium Customers). The diagram below represents audience similarity on a spectrum of model performance.

Consider the following examples of audiences with different levels of similarity.

• Overlapping Audiences (Bad)
  • audience multi-year subscriber → audience with high loyalty
  • unknown audience → audience with no purchases
• Divergent Audiences (Bad)
  • unknown audience → audience with newsletter signup and single purchase
  • anonymous audience → audience with multiple purchases
• Adjacent Audiences (Good)
  • unknown audience → audience with email
  • known audience of single purchases → known audience with multiple purchases

If you’ve selected divergent or overlapping audiences in your model, you will see diagnostic messages to guide you to select audiences that are more adjacent.

Most Predictive Audiences are built by identifying users in the source audience who have a high model score. This "high score" is called the Decision Threshold, and in most cases is 0.5 To adjust the reach of the audience you’re building, you might consider using a different decision threshold.

• Use a lower decision threshold to reach more users in the source audience.
• Use a higher decision threshold to be more accurate but reach less users in the source audience.

Diagnostic messages may suggest creating audiences using different decision thresholds, but you are always free to make the decision threshold whatever makes the most sense to generate a Predictive Audience of the right size.

Most of the time, diagnostic messages for unhealthy models will suggest building a model with different source or target audiences. Occasionally, you might see a diagnostic message suggesting that a model should be built with different features.

The model could not find sufficient signal in the features provided. Try either using Auto Tune or providing additional features.

In these cases, the underlying model didn't have enough of a signal in the data provided to be able to predict target behavior.

Imagine that you're an online retailer trying to build Lookalike Models for multi-purchasers, but you build a model that doesn't have any purchase data. Or you want to build an email churn model, but you don't have any email data in your model. In these cases, regardless of how your source and target audiences are constructed, your model would be missing the underlying signal required to make a successful model.

You could either manually select additional fields to include in the Model Builder, or you can use the Auto Tune option on the model to have the machine attempt to automatically identify the best field candidates to include in the model.