User-Data Onboarding by CSV

While Dynamic Yield segmentation capabilities are extensive, you can enrich them even further by onboarding additional user data (such as CRM). This enables you to target experiences and segment users according to attributes that are unique to your business (for example, a particular VIP status). 

You can upload batch data, nightly, in CSV format, as described in this article; or, you can use our on-demand User Data API. The API is the best choice when you need real-time data activation, and the number of users that need to be updated is not large.

Batch upload

How it works:

1. A user data file is uploaded on a daily basis, which includes a list of users (with an identifier) and their attributes (such as age group).
2. After the user data file is uploaded, you can see new audience conditions based on the attributes included in the file.
3. When a user visits your site and identifies (even once, and even before the user data file is onboarded), Dynamic Yield connects the user identity in the file to the device they are browsing from. If they match audience conditions based on these attributes, they are included in the audience.

Onboarding process

The data onboarding process is done by your Technical Account Manager, but it requires data preparation on your side. This is the process:

Step 1: Choose the attributes to onboard

We recommend starting with 5 top attributes (you can always increase this number, up to 50). Focus on the attributes that are relevant for segmentation and are not supported natively in the Dynamic Yield platform. For example:

• • Demographic: Gender, Age group, Marital status, #of children, Education, Industry, Job Title
  • Geographic: Billing country, Shipping country
  • Interests and preferences: Favorite category, Language
  • Purchasing habits: Average order value, Items per order, Days since last order, Average inter-purchase time, Lifetime Orders, Lifetime Net Revenue, Recency, Number of Orders, Net Merchandize Value 

Step 2: Prepare a metadata attribute file

This file instructs Dynamic Yield how to transform your attributes into user-friendly audience conditions. For example, which condition requires a date type, and which requires a number (with "more than" and "less than" capabilities). This step is quite technical, and your Technical Account Manager will assist you in creating this file.

This is the file structure:

 

Column name	Description
Field_Name	The technical attribute name. This does not appear in the Dynamic Yield user interface.
Display_Field_Name	The name used as the condition name in the Dynamic Yield user interface.
Type	The condition type: Int/double/long: A number (with "more than", "less than", and "exactly" capabilities)
	String: A string of text
	Enum: A dropdown option
	Boolean: A value that is either true or false. Supported values: “True” : "true", "1", "y", "yes", "Y" “False” :"false", "0", "n", "no", "N"
	Date: A value that represents a specific date in a simpleDateFormat
	Array: A list of values, separated by a comma (,), pipe (|), semi-colon (;) or tab. Note that it must be a different separator than the one used to separate the columns.
	Bucket: Groups of lists of values (for example, group age values of "30", "34", and "35" into a single value of "30-35")
Dropdown_Source_Value	The technical value name. This does not appear in the Dynamic Yield interface.
Dropdown_Display_Value	The name that is used as the condition value in the Dynamic Yield interface.

 

Step 3: Determine a common identifier

Each record of user data should include a unique identifier, in order to match between a first-party data user and an online user. The recommended unique identifier is the email address (SHA256 hash of the lowercase email address). For privacy reasons, do not use plain text PII (such as name, phone number, or email address) as the identifier.

Make sure that the Identify, Login, Signup, and Newsletter Subscription events use the same identifier.

Optional: Any additional identification instance by the visitor (for example, arriving from a newsletter) can be reported to Dynamic Yield with the Script Utilities.

Step 4: Provide a sample file 

Provide your Technical Account Manager with a representative sample of your first-party data in CSV format. For example:

Step 4: Initiate a daily upload process

Note: if your file exceeds 40 attributes or 100MB, you must split it into multiple files. The following are instructions for large file handling:

1. Get credentials to a AWS S3 bucket from your Technical Account Manager to gain access to a dedicated folder to which you can upload your first-party user data. 

   For example, the path might look like: s3://com.dy-external.crm/Bucket_Id/{Folder name} OR s3://eu-central-1.dy-external.crm/Bucket_Id/{Folder name} for EU sections.

   AWS regions where the buckets are created:
   • US East (N. Virginia - us-east-1)
   • Europe (Frankfurt - eu-central-1)
2. Set up an upload process that syncs the data in your dedicated folder to guarantee that your data is up to date. Use the following guidelines:
   • File type: CSV.  

   • File format: UTF-8 encoded. If using non-latin characters (Russian, Japanese, and so on) then the encoding of the CSV file should be UTF-8 with BOM.

   • Filename: CRM_data.csv
   • Upload frequency: Daily.
   • Each daily upload should be added to a folder that has the following name format: “upload_YYYY-MM-DD_HH-mm” (upload_2021-02-28_00-03).
   • The daily data dump can include only:
     • One file.
     • A full snapshot of your first-party data. Python Example:
       

       id = "replace this with your access key"
       secret = "replace with your secret key"
        
       if __name__ == '__main__':
           import boto
           s3_connection = boto.connect_s3(aws_access_key_id=id, aws_secret_access_key=secret, is_secure=False)
           bucket = s3_connection.get_bucket('com.dynamicyield.crm', validate=False)
           
          
           key = bucket.get_key('/test', validate=False)
           key.set_contents_from_string('test content')
           print key.get_contents_as_string()
           
           rs = bucket.list(prefix='/')
           for i in rs:
               print i
               

Once these steps are complete, the data is processed by the Dynamic Yield batch processes. Upon completion, the relevant conditions appear in your account. Data processing time can vary. Follow up with your Technical Account Manager to find out when to expect your CRM Data conditions to be available.

Large file handling

The maximum supported file size is 100 MB. Larger files should be split into multiple files, each around 40 MB. The structure of the files must be identical, and uploaded to the same daily folder. However, each file should contain information about different users.  

When working with one file, the name should be CRM_data.csv. However, when working with multiple files, add a number to the beginning of the name as follows:

• part_0_CRM_data.csv
• part_1_CRM_data.csv
• part_2_CRM_data.csv
• ...
• CRM_data.csv (last file that triggers the ingestion)

All file names should end with CRM_data.csv.

Each file should include the headers.

For more information, speak to your Customer Success Manager.

Monitoring onboarding

You can always see the status of the onboarding in the Data Feed screen. The feed type is "User Data".

The  Last Successful Sync column shows you when was the last time that we replaced the data with a newer version. 

The Feed Status column shows you the status of the last sync, and if there are issues, you can download an error log to fix and debug them. Statuses include:

• Synced: No issues were found.
• Synced with warnings: The file was synced, even though some issues were found (for example, some users appear in multiple rows, or some values are not expected or missing). It is recommended to review any issues and resolve them.
• Failed: The file was not synced, because major issues were found (for example, the column names are not as defined in the schema, or there is no user identifier column). In this case, the file is not replaced at all, and user data is based on the previously synced file.

Dashboard Alerts

Alerts appear on the dashboard if:

• The last sync failed, and a new successful attempt was not done in the past day. A link to the error log is provided to help you fix any issues.
• The last sync was successful, but it occurred more than 10 days ago. Updating the feed on a daily basis is recommended to ensure user data is up to date.

Creating audiences based on first-party data

Once your data has been onboarded, you can immediately use the Audience Explorer to explore these valuable segments and analyze their performance in comparison to your entire online user base, complemented with users’ behavioral data captured by Dynamic Yield. These conditions can be coupled with any other Dynamic Yield condition available in your account and used to create and save micro-targetable segments (audiences) that can be incorporated into your Dynamic Yield Site Personalization experiences and Recommendations.