Update Your User Data Feed by API

Follow

The User Data API enables you to update your user data on demand, including partial updates.      

With the User Data API, you can use data that was collected outside of Dynamic Yield for segmentation and targeting, in real time, and with reduced load compared to the file-based feed. Also, you can run updates, view results, and monitor activity with no need for assistance from your account manager.

When should you choose the API rather than the CSV method?

You can experiment with calling the API, checking the status in the log and fine-tuning it to match your business needs.

Prefer this method when:

  • User data is needed in real time (for personalizing the user's experience, for example, letting a user know that they have loyalty points that are set to expire).
  • Only a small number of users need an update (with the CSV method, the entire dataset is uploaded. With the API, you can update only specific new data).

User-data onboarding flow

Setting up and using the API includes the following steps. Some preparation is done on your side (by either a business user or a developer), and some are done in the Dynamic Yield console, as indicated.

Follow these instructions if you are setting up user-data onboarding for the first time. If you are migrating from CSV upload to API, see Existing accounts: Moving from CSV to API.

Screen_Shot_2021-11-02_at_13.27.14.png

Preparation:

  1. Decide which fields (columns) to pass in the API (name, display, type, and so on) (business user). 

Pro tip:

All user attributes must be sent in each API call. You can streamline the process by grouping the columns into User Data feeds based on use cases, for example, based on:

  • Frequency: Group together fields that you plan to update with the same frequency (eliminates needless updates). 
  • Attribute availability: All attributes that are updated in the feed must be available for each API call, so separate those that change infrequently from those that change often.  
  • Number of users to update: For a small number of user updates, the API is best-suited. If you need to do a full update, use the API load or a CSV file method.
  1. Generate an API key (if you haven't already done so).

Setup in the Dynamic Yield console (developer):

  1. Set up the feed:
    1. Create a new feed with the required settings.
    2. Create the JSON schema. You can build the schema from scratch, or start with the sample file provided in the console, and edit it as needed. 
    3. Set the sections that share the data, and save the draft. The schema is available within 5 minutes, and the API can now be tested. 
    4. Set up the code for your API calls. 

Test the feed:

Note: This is an important step, because after activation, the schema can be changed only in limited ways. The test user data is not committed, and campaigns do not yet work.

  1. Call the API you coded in the previous step.
  2. The API call returns a transaction ID in a field called dy_trace_id. Using this ID, go to the API log and search for the API call status.
  3. In the log, ensure that the API request status is correct (202). Then, check that no errors exist for the nested objects – the API call status that is returned doesn't include them.
    image__2_.png
    When the API is called, a status is returned. This status indicates the call is received. In the log, you can see the details of any errors for each action, relating to specific objects they include. 

Onboard user data: 

  1. Activate the feed (developer).
  2. Call APIs in accordance with your business flow.
  3. Monitor the feed (business user and developer). Use:
    • Alerts: Provide information about issues with the API call and its included objects. 
    • Logs: See Experience APIs Logs for more information. The User Data API logs behave the same.

Schema guidelines

The schema is a table that determines where and how the information is stored in Dynamic Yield. This information is provided in JSON syntax, and copied into the Dynamic Yield console. 

We've created a tool to make creating and modifying your JSON schema easier:

JSON Schema Rules

Goal

The purpose of the schema is:
  • Define the table where the data is stored, so values can be validated when they are sent via the API or CSV.
  • Define how the Dynamic Yield console shows the condition (when defining an audience based on criteria).

Size limitation

The JSON definition schema cannot exceed 64K.

Fields

Each field is stored as an object within the fields array. The JSON schema must include at least 1 object in the field array, with the following parameters, as shown in this example: 
"fields": [
   {
      "name": "purchase_count",→ field name, sent in the API or column name in the CSV
      "display": "Purchase Count", → the condition name, that appears in DY admin
      "type": [
        "null", → is the object nullable or not
        "int" → nature of the field passed
      ],
      "uiType": "number" → nature of the field that appears in DY admin
    }
]

Additional values in the layout

If you want create display values in the Dynamic Yield console in addition to those set in the schema: 

  • Create a UI object that contains its definition (for example, map “true” to “t”).
  • Parallel to the “field” array, add a "uiTypes" object.
  • Map that UI object using the existing field object.
  • Verify that each UI object in the field arrays is defined.

Example of what this can look like:

Screen_Shot_2021-11-03_at_13.40.53.png

Sample code:


      "fields": [
{

      "name": "has_offer",
      "display": "Has Offer",
      "type": [
        "null",
        "boolean"
      ],
      "uiType": "ref",
      "uiTypeRef": "UIhas_offer"
    }
],

"uiTypes": {
    "UIhas_offer": {
      "uiType": "select",
      "validation": {
        "values": [
          {
            "name": "Yes",
            "value": "true"
          },
          {
            "name": "No",
            "value": "false"
          }
        ]
      }
    }
Editing an active JSON schema
You can edit an active schema according to the following rules and limitations:
You can:
  • Add or remove values, as long as they are not in use in an audience conditions
  • Change display names of fields
  • Add new fields:
    • Add only at the end of the existing schema.
    • Fields must be nullable (non-mandatory) (must be of type “null”).
    • Must include "default": null parameter.

You can't:

  • Delete fields.
    Pro tip: Instead, change a field to non-mandatory and stop updating it.
  • Change the order of fields.
  • Change field types.
  • Remove “null” from a field type, as it will make the field mandatory.
Note that when you save the edited schema, your changes apply immediately. Make sure to update your API calls accordingly.
Sample JSON schema
This object includes several column types that can be referenced. Although the JSON format doesn’t support comments, this example includes some "fake" comment lines, beginning with "//" followed by the comment text. If you decide to use these objects, make sure you first remove these lines. 
{
"fields": [
 {
 "//":"Field with values example - Referenced in uiType with the possible values",
 "name": "VIPPersona",
 "display": "VIP Persona",
 "type": ["null", "int"],
 "uiType": "ref",
 "uiTypeRef": "UIMetalVIP"
 },
 {
 "//":"Number example - no need to reference in uiType",
 "name": "CustomerScore",
 "display": "Customer Score",
 "type": ["null", "int"],
 "default": null,
 "uiType": "number"
 }, 
 {
 "//":"string example - no need to reference in uiType",
 "display": "Account Manager",
 "name": "AccountManager",
 "type": [
 "null",
 "string"
 ],
 "uiType": "input"
 },{ 
 "//":"Boolean example - no need to reference in uiType",
 "name": "IsSubscribedTo",
 "display": "Is Subscribed To Emails",
 "type": [
 "null",
 "boolean"
 ],
 "uiType": "ref",
 "uiTypeRef": "UIIsSubscribed"
 },
 {
 "//":"Date example - no need to reference in uiType",
 "name": "AnniversaryGiftExpires",
 "display": "Anniversary Gift Expires",
 "type": [
 "null",
 "long"
 ],
 "uiType": "date"
 }
 ],
 "uiTypes": {
 "UIMetalVIP": {
 "//":"Example how to represent different display values in the Dynamic Yield console",
 "uiType": "select",
 "validation": {
 "values": [{
 "name": "VIP Gold",
 "value": 0,
 "source": "goldVIP"
 }, {
 "name": "Other",
 "value": 1,
 "source": "otherVip"
 }, {
 "name": "VIP Silver",
 "value": 4,
 "source": "vipSilver"
 }
 
 ]
 }
 },
 "UIIsSubscribed": {
 "//":"Admin vallues: boolean",
 "uiType": "select",
 "validation": {
 "values": [
 {
 "name": "No",
 "value": "false"
 },
 {
 "name": "Yes",
 "value": "true"
 }
 ]
 }
 }
 }
}

Use our User Data Schema Generator tool to create or edit your schemas.

Setup

Generate an API key

An API Key is used to identify the source of API calls that are sent to Dynamic Yield. You must include the API Key in all of the API calls specified below. To generate an API Key:

  1. Go to Settings › API Keys and click New Key. 
  2. Specify User Feed for the ACL value and click Save. 
  3. Copy the value of the key to a secure location. 
    Screen_Shot_2021-12-07_at_10.15.33.png
  4. The new key now appears in your list.
    Screen_Shot_2021-12-07_at_10.24.20.png

Create a new feed

  1. Go to Assets › Data Feeds and create a new User Data feed.
    Screen_Shot_2020-02-11_at_9.19.10.png
  2. For the Feed Source, select Sync via API.
  3. Enter the schema for the feed.
  4. Define the CUID type. Default is email, so you need to change this value only if you want to use a different ID type.
  5. Click Save as Draft.

Existing accounts: Moving from CSV to API

If you are already onboarding user data using the CSV upload method and want to move to the User Data API:  

  1. Open the User Data feed.
  2. Change the feed settings to Sync via API. Note that schema changes are not supported at this time.
  3. Call APIs in accordance with your business flow.
  4. Monitor the feed (business user and developer). Use:
    • Alerts: Provide information about issues with the API call and its included objects. 
    • Logs: See Experience APIs Logs for more information. The User Data API logs behave the same.
  5. Activate 

Note: Existing user attributes is retained when switched to API. 

Important: Once you switch to the API sync, you cannot go back to the CSV method without assistance from technical support, so make sure this is what you want to do. 

Creating API calls

Actions include: 

  • Upsert: Create a new trait row for a user, or override an existing one. Only the sent traits are saved, the rest are "null". 
  • Delete: Delete an existing entry.

Call structure 

US data center:

https://dy-api.com/v2/userdata/{feedkey}/bulk

European data center: 

https://dy-api.eu/v2/userdata/{feedkey}/bulk

All calls must include:

  • feedkey: (string) The data feed ID. You can get the ID of an existing data feed by going to Assets › Data Feeds and looking in the ID column. If you don't see the ID column, you can add it using the Column dropdown. 
HTTP Method
  • POST
Headers
  • DY-API-key: Your API Key
  • Content-Type: application/json
Parameters
  • feedId: (string) The ID of the feed
Body
  • CUID
  • CUID type
  • Time stamp
  • Data (for upsert only)
  • Action to perform

Body example

{
  "requests":[ 
    {
    "action": "upsert",
    "cuid": "myuser123",
    "cuidtype": "ihe",
    "timestamp": "2021-07-24T13:44:17+02:00",
    "data": {
         "VIP": "T",
         "contacts_buyer": "T",
         "premium_score": "0.8",
         "segment_name": null,
         "last_visit": "2020-05-16T05:50:06-05:00",
            }
    },

    {
    "action": "upsert",
    "cuid": "myuser456",
    "cuidtype": "ihe",
    "timestamp": "2021-07-24T13:45:52+02:00",
    "data": {
         "VIP": "F",
         "contacts_buyer": "T",
         "premium_score": "0.3",
            }
    },

    {
    "action": "delete",
    "cuid": "myuser789",
    "cuidtype": "ihe",
    "timestamp": "2022-05-24T12:31:09+02:00",
    }
  ]
}

Response example

{
    "error": null,
    "dy_trace_id": "eed3b778281fe4bd435902c8c7f854a3"
}

If an error occurs, the response looks like this:

{
    "error": {
       "message": "The feedKey must match the feed key in the feed settings",
       "type": "E019"
    },
    "dy_trace_id": "74eb317b0a44addc30e765e6b9a83675"
}

Creating audiences based on user data

When your data is onboarded, you can immediately use the Audience Explorer to explore these valuable segments and analyze their performance in comparison to your entire online user base, complemented by users’ behavioral data captured by Dynamic Yield. The new condition appears under User Properties using the User-Data feed name, and it can be coupled with any other Dynamic Yield condition available in your account and used to create and save micro-targetable segments (audiences) that can be incorporated into your Dynamic Yield Site Personalization experiences and Recommendations.

2017-03-29_11-43-25.gif

Troubleshooting response errors

API error messages

Possible errors that can be returned include:

  • 202: request accepted - trace ID returned 
  • 400: bad request/error parsing request
  • 401: invalid or missing API key
  • 403: access denied for a valid API key (wrong section, wrong feed id, illegal operation)
  • 405: wrong HTTP method (not POST)
  • 422: exceeded feed size limitation/exceeded number of actions per batch
  • 429: too many requests received
  • 451: wrong key type used (server-side key from client side, or client-side key from server-side)
  • 500: unspecified internal error

Prerequisites and limitations

  • API-based feed cannot be changed to CSV method.
  • Each site must have a unique API key.
  • API request limit: 
    • Ability to update up to 100 users in 1 API call
    • Up to 100 API calls in 1 second, adding up to 600K updates per minute