Developing with Lytics / Lytics API Documentation

Data Upload

APIs for collecting or uploading data. There are two main APIs, one for uploading large files (bulk, millions of records) and another for sending real-time, or slightly batched events (Less than 10MB).

stream: A stream name is like a table in database, each stream is a differnt Type of data. You assign a name to your data uploads to identify each stream/type. However, it should be consistent across uploads of same type of data.

Timestamp Fields: When uploading historical, or event data you may have a time/date at which an event occured. Normal event uploads get an event timestamp at the server time but you can over-ride this behavior so the timestamp is defined in the event you upload. This is a timestamp_field parameter in api's below. The formatting for this is very, very lenient with over 50 format's supported.

Bulk Upload Considerations: When considering large uploads to the Lytics platform, be aware that this may cause an event backlog on your account. This occurs as our platform queues incoming information as it processes the content of the upload. Processing begins when the upload completes and can last considerably longer than the initial upload. This can result in a delay in seeing information from other sources in our platform as well as delays in any outbound, trigger-based workflows.

Bulk CSV Upload

/collect/bulk/{stream}{?dryrun,timestamp_field,headers,delimiter,append,droperrors,filename}

Upload a large collection of data via csv, or JSON files.

Caution: Our csv parser is very strict, with escaping quotes etc being less lenient than many parsers you may have worked with. This is to ensure high quality data.

Error Handling Upon error, it will stop processing and return. So, we suggest trying uploading files with dryrun=true to verify your files will be accepted. If a few rows have issue you might try droperrors to see how many it is.

Parameters
streamstring (required)
Ex: user_db
The DataType aka Table of type of data being uploaded. See Data, Streams in web admin.
dryrunboolean (optional)
Default: false
timestamp_fieldstring (optional)
The name of the column or field in file that contains event timestamp.
headersstring (optional)
CSV only, Comma delimited list of header columns for csv (if not first row)
delimiterstring (optional)
CSV only, if not the default comma, either t=tab, or | may be accepted
appendstring (optional)
CSV only, JSON encoded object of additional name-value pairs to append to each record.
droperrorsboolean (optional)
Default: false
filenamestring (optional)
Ex: "myfile.csv"
Just for record-keeping in our event - stream.
POST

Bulk CSV Upload

/collect/bulk/{stream}{?dryrun,timestamp_field,headers,delimiter,append,droperrors,filename}

Upload CSV files (may be gzipped). They MUST contain headers: either in first row (by default) OR in headers querystring field. If the file contains a timestamp field you want us to use see timestamp_field querystring parameter for how we extract it.

Note Domain this api is on a different domain https://bulk.lytics.io because the api.lytics.io domain is limited to 10MB file uploads.

# the "user_regdata" is the STREAM (ie Type) of data, if there 
# is multiple different types of data (userdata, emaildata, events) 
# they would have different streams.

# user data
export LIOKEY="{API Token}"
curl -s -H "Authorization: $LIOKEY" \
  -H 'Content-type: application/csv' \
  --data-binary @users.csv \
  "https://bulk.lytics.io/collect/bulk/user_regdata?timestamp_field=register_date" \
 | jq '.'


# only do a dry-run to see if it is formatted correctly
curl -s "https://bulk.lytics.io/collect/bulk/user_regdata?access_token=$LIOKEY&timestamp_field=register_date&dryrun=true" \
    --data-binary @users.csv -H 'Content-type: application/csv' | jq '.'

# append fields to each row
#  where   append = urlencode({"source":"crm"})
#  often used when file-name contains information

curl -s "https://bulk.lytics.io/collect/bulk/user_regdata?append=%7B%22source%22%3A%22crm%22%7D&access_token=$LIOKEY" \
    --data-binary @users.csv -H 'Content-type: application/csv' | jq '.'


# full example uploading gzipped file

echo 'user_id,email,reg_date,item_count
7456,[email protected],"2012-10-17T17:29:39.738Z",82
1234,[email protected],"2012-12-11T19:53:31.547Z",82' > users.csv

gzip users.csv

curl -s "https://bulk.lytics.io/collect/bulk/user_regdata?access_token=$LIOKEY&timestamp_field=reg_date&dryrun=true&filename=users.csv" \
    --data-binary @users.csv.gz \
    -H 'Content-type: application/csv' | jq '.'

Request

Headers
Content-Type: application/csv
Body
name,id,registerd_date
jane,123,10/21/2004

Response 200

Headers
Content-Type: application/json
Body
{
    "status": 200,
    "message": "success",
    "data": {
        "message_count": 2,
        "rejected_count": 0,
        "content-type":"application/json",
        "droperrors": false,
        "dryrun": false,
        "timestamp_field": "event_created",
        "filename": "myfile.csv"
    }
}

Bulk JSON Upload

/collect/bulk/{stream}{?dryrun,timestamp_field,filename}

JSON New Line Delimited file uploads. Each line contains a single record. The only limit is that nested array/objects are not allowed (see examples below).

Note Domain this api is on a different domain https://bulk.lytics.io because the api.lytics.io domain is limited to 10MB file uploads.

Parameters
streamstring (required)
Ex: user_db
The DataType, or "Table" of type of data being uploaded
dryrunboolean (optional)
Default: false
timestamp_fieldstring (optional)
The name of the column or field in file that contains event timestamp.
filenamestring (optional)
Ex: "myfile.json"
Just for record-keeping in our event - stream.
POST

Bulk JSON Upload

/collect/bulk/{stream}{?dryrun,timestamp_field,filename}

JSON formatting info.

This JSON is ok:

{"order_ids":[1,2,3]}

This is not valid because it has objects nested inside arrays:

{"orders":[{"id":1,"price":22},{"id":2,"price":23}]}

Request

Headers
Content-Type: application/json
Body
{"event":"register","date":"2014/04/05"}
{"event":"login","date":"2014/04/05"}

Response 200

Headers
Content-Type: application/json
Body
{
    "status": 200,
    "message": "success",
    "data": {
        "message_count": 2,
        "rejected_count": 0,
        "content-type":"application/json",
        "droperrors": false,
        "dryrun": false,
        "timestamp_field": "event_created",
        "filename": "myfile.json"
    }
}

Data Upload

/collect/json/{stream}{?dryrun,timestamp_field,filename}

Upload event(s) to API. Unlike the bulk API above, this only accepts valid JSON (not new line delimitted) and it accepts arrays (lists) of events or a single event. Can be used to upload changes to user-attributes, or event records. Limited to 10MB.

# Example for multiple events
export LIOKEY="{API Token}"
curl -s -H "Authorization: $LIOKEY" \
  -H 'Content-type: application/json' \
  -XPOST "https://api.lytics.io/collect/json/crm_events?timestamp_field=date" \
-d'
[
    {"event":"registered","date":"10/03/2013","user_id":"1234","tags":["a","b","c"]}
    , {"event":"purchased","date":"10/03/2013","user_id":"456","price":123.45,"product_id":"123456"}
]
' | jq '.'


# example single record
curl -s -XPOST "https://api.lytics.io/collect/json/user_db?access_token=$LIOKEY" \
    -H 'Content-type: application/json' -d'
{"user_id":"1234","tags":["a","b","c"]}

' | jq '.'
Parameters
streamstring (required)
Ex: user_db
The DataType, or "Table" of type of data being uploaded
dryrunbool (optional)
Default: false
timestamp_fieldstring (optional)
The name of the column or field in file that contains event timestamp.
filenamestring (optional)
Ex: "myfile.json"
Just for record-keeping in our event - stream.
POST

Data JSON Upload

/collect/json/{stream}{?dryrun,timestamp_field,filename}

Request

Headers
Content-Type: application/json
Body
[{"event":"register","date":"2014/04/05"},
  {"event":"login","date"}]

Response 200

Headers
Content-Type: application/json
Body
{
  "status": 200,
  "message": "success",
  "data": {
    "message_count": 2,
    "rejected_count": 0,
    "content-type":"application/json",
    "droperrors": false,
    "dryrun": false,
    "timestamp_field": "event_created",
    "filename": "myfile.json"
  }
}