Product Feeds

Follow

A Product Feed is a file that contains your product catalog (all your products, with product metadata). It powers affinity-based personalization, product recommendations, social proof messaging, and more. 

You can create product feeds by syncing data feed files as specified in this article or create and sync a product feed using APIs. 

Creating a Product Feed Overview

Syncing your product catalog with Dynamic Yield includes the following steps (clicking the numbered links will navigate you to that portion of this article):

  1. Create the file with all the required product metadata (e.g. price, SKU). This is done on your end. 
  2. Sync the file with Dynamic Yield by referring to a URL, uploading it to a secured S3 bucket directly, or uploading it to S3 via SFTP.  To sync the feed using APIs, see Creating and Sync a Product Feed using APIs.
  3. Parse the source file, if needed, by writing a JavaScript code that manipulates the source file to adjust it to the format required by Dynamic Yield (e.g. changing attribute names to match to Dynamic Yield requirements). This is done in the Product Feed advanced settings, and is only supported for feeds that are smaller than 200,000 items or 180MB.
  4. Save and preview to check if the feed meets all Dynamic Yield requirements. Afterwards you can save and publish. 

Note: If your feed is larger than 20,000 items, please contact your Customer Success Manager. The syncing process (Steps 2-4) will be done by them. The total feed size is limited to 5 Million items. 

 

1. Create the File

Before you can upload and sync a Product Feed to Dynamic Yield, you need to make sure your feed file complies with size, format, and content requirements.

File Type and Size

Feed Size File Type
Up to 20,000 items CSV, JSON, XML
20,000-200,000 CSV, JSON
More than 200,000 CSV

It can be stored in any URL or a local folder, or in an Amazon S3 bucket that we provide. You can transfer the feeds to Amazon S3 using SFTP as well.

If you use a CSV file, items must be separated by commas.

Product Feed Structure

The Product Feed must include the basic information about your product (SKU, group ID, price, image, URL, if it's in stock or not, and its categories). You can see the list of mandatory attributes below, with detailed instructions (format, limitations, etc.):

Column (case sensitive) Description Format Example
 sku Product Unique Identifier. Note: SKUs cannot contain spaces or the string "//" String
(up to 256 characters, no spaces) 		  111
group_id Identifies a group of products that differ in some product attributes. Note: this value cannot contain spaces. String
(up to 256 characters, no spaces) 		  22
name Product Name  String 
(up to 1,000 characters) 		  t_shirt
 url URL to the product details page (must be a valid URL, starting with HTTP/HTTPS) String 
(up to 1,000 characters)		  http://mystore.com/111
 price Price of the product (must be a number)  Float 
(up to 1,000 characters)		 9.50 
 in_stock  Indication whether the product is in stock (noted by ‘true/false’) Boolean 
(up to 1,000 characters)		 true
 image_url The URL to the product image (must be a valid URL, starting with HTTP/HTTPS) String 
(up to 1,000 characters)		 http://mystore.com/111.png 
categories  The categories associated with the product from general to specific Strings
(separated by pipes with no spaces, up to 1,000 characters)		 Women’s|Trousers
& Jeans|Chinos
keywords (optional but strongly recommended) Any additional information describing the product, separated by pipes. Used for our machine learning and affinity algorithms. Because this column is optional, blank values will not trigger errors or warnings. String 
(up to 1,000 characters)		 sale_item

All of the columns listed above, except "keywords", are mandatory. If a product is missing a value in one of these columns, it will be skipped. If more than 10% of products have an empty value in one of the columns, the entire sync will fail.

Note: If your feed source has this information, but names are different (for example, if you have a Google Product Feed), or this information exists in 2 separate feeds - you can use a parser function when syncing the feed to adapt it according to the guidelines above. This is available for feeds that have less than 200,000 items and 180MB.

Custom Columns

You can also add more columns to the feed that describe your product. These can later be used for targeting (e.g. if you add "color", you can use it to target users who purchased blue items), affinity score (e.g. include color affinity in the affinity recommendation algorithm), define merchandising rules in your recommendation widgets (e.g. never recommend products with a specific brand). 

Guidelines:

  • Custom column values must be string, up to 1,000 characters, with no special characters.  Special characters may as long as they are wrapped within a string. 
  • Custom column names must not start with 'id'.
  • Once you add a column, you cannot remove it. You can stop using it or return empty values, but the column must be in any future sync.

*You can create a total of up to 300 columns when working with multiple languages (including all mandatory, custom, and translated columns).

Multi-Language Support

To support multiple languages, you can specify different values for different languages for any field. Use the format “lng:<language code>:<column name>”: “<value>”. For example, if wanted to have a translation for a product name, you would create two rows for the product as follows:

"lng:en_EN:name":"White Pants"
"lng:de_DE:name":"weiße Hosen"

The language code should match the name you use in the page context. For details, see Page Context. The language code can have up to two (_) or (-) between the values. For example: "lng:en_EN_X".

Because these columns are optional, blank values will not trigger errors or warnings.

You can create a total of up to 300 columns when working with multiple languages (including all mandatory, custom, and translated columns).

To learn how to target campaigns based on user locale (using the "lng" attribute), see the Multilingual Support article.

Note: While Dynamic Yield supports feeds of up to 5 million items, there are some limitations as the size of the feed increases. Additionally, larger feeds will experience slower sync times as well as slower recommendation serving speed.

Product Feed File Examples

XML Product Feed Example 1
 <items>
  <item>
   <sku>111</sku>   
   <group_id>22</group_id>
   <price>9.5</price> 
   <categories>mens|sport</categories>
   <in_stock>true</in_stock> 
   <name>t_shirt</name> 
   <image_url>http://dy.com/shop/111/ex.jpg</image_url>
   <url>https://www.dy.com/shop/111.html?ch=22</url>
  </item>
</items>
XML Product Feed Example 2
<items>
 <item>
 <sku>1125-blue-42</sku>
 <group_id>1125</group_id>
 <categories>shoes|mens|road</categories>
 <name>Desert Eagle</name>
 <url>https://www.shop.com/en_US/products/desert-eagle</url>
 <price>225</price>
 <image_url>//images/en_US/shoe.png</image_url>
 <in_stock>true</in_stock>
 <keywords>active|training|black|grey|direct|neutral</keywords>
 <display_price>225$</display_price>
 <gender>mens</gender>
 <product>shoes</product>
 <surface>road</surface>
 <best_for_1>active</best_for_1>
 <best_for_2>training</best_for_2>
 <color_1>black</color_1>
 <color_2>grey</color_2>
 <cushioning>direct</cushioning>
 <support>neutral</support>
 <lng:en_US:name>Desert Eagle</lng:en_US:name>
 <lng:en_US:in_stock>true</lng:en_US:in_stock>
 <lng:en_US:url>https://www.shop.com/en_US/products/desert-eagle</lng:en_US:url>
 <lng:en_US:image_url>//images/en_US/shoe.png</lng:en_US:image_url>
 <lng:en_US:display_price>200$</lng:en_US:display_price>
 <lng:en_GB:name>Desert Eagle</lng:en_GB:name>
 <lng:en_GB:in_stock>false</lng:en_GB:in_stock>
 <lng:en_GB:url>https://www.shop.com/en_GB/products/desert-eagle</lng:en_GB:url>
 <lng:en_GB:image_url>//images/en_GB/shoe.png</lng:en_GB:image_url>
 <lng:en_GB:display_price>200£</lng:en_GB:display_price>
 <lng:fr_FR:name>Aigle du désert</lng:fr_FR:name>
 <lng:fr_FR:in_stock>true</lng:fr_FR:in_stock>
 <lng:fr_FR:url>https://www.shop.com/fr_FR/products/desert-eagle</lng:fr_FR:url>
 <lng:fr_FR:image_url>//images/fr_FR/shoe.png</lng:fr_FR:image_url>
 <lng:fr_FR:display_price>200€</lng:fr_FR:display_price>
 <lng:de_DE:name>Wüstenadler</lng:de_DE:name>
 <lng:de_DE:in_stock>true</lng:de_DE:in_stock>
 <lng:de_DE:url>https://www.shop.com/de_DE/products/desert-eagle</lng:de_DE:url>
 <lng:de_DE:image_url>//images/de_DE/shoe.png</lng:de_DE:image_url>
 <lng:de_DE:display_price>200€</lng:de_DE:display_price>
 </item>
</items>
JSON Product Feed Example 1
[{
"sku": "111",
"group_id": "22",
"price": 9.50,
"name": "t_shirt",
"in_stock": true,
"image_url": "http://dy.com/shop/111/ex.jpg",
"url": "https://www.dy.com/shop/111.html?ch=22",
"categories": "mens|sport"
}
{
"sku": "123",
"group_id": "22",
"price": 8.50,
"name": "t_shirt",
"in_stock": true,
"image_url": "http://dy.com/shop/123/ex.jpg",
"url": "https://www.dy.com/shop/123.html?ch=22",
"categories": "mens|sport"
}
]
JSON Product Feed Example 2
[{
   "sku": "1125-blue-42",
   "group_id": "1125",
   "categories": "shoes|mens|road",
   "name": "Desert Eagle",
   "url": "https://www.shop.com/en_US/products/desert-eagle",
   "price": 225.0,
   "image_url": "//images/en_US/shoe.png",
   "in_stock": true,
   "keywords": "active|training|black|grey|direct|neutral",
   "display_price":"225$",
   "gender": "mens",
   "product": "shoes",
   "surface": "road",
   "best_for_1": "active",
   "best_for_2": "training",
   "color_1": "black",
   "color_2": "grey",
   "cushioning": "direct",
   "support": "neutral",
   "lng:en_US:name":"Desert Eagle",
   "lng:en_US:in_stock": true,
   "lng:en_US:url": "https://www.shop.com/en_US/products/desert-eagle",
   "lng:en_US:image_url": "//images/en_US/shoe.png",
   "lng:en_US:display_price":"200$",
   "lng:en_GB:name":"Desert Eagle",
   "lng:en_GB:in_stock": false,
   "lng:en_GB:url": "https://www.shop.com/en_GB/products/desert-eagle",
   "lng:en_GB:image_url": "//images/en_GB/shoe.png",
   "lng:en_GB:display_price":"200£",
   "lng:fr_FR:name":"Aigle du désert",
   "lng:fr_FR:in_stock": true,
   "lng:fr_FR:url": "https://www.shop.com/fr_FR/products/desert-eagle",
   "lng:fr_FR:image_url": "//images/fr_FR/shoe.png",
   "lng:fr_FR:display_price":"200€",
   "lng:de_DE:name":"Wüstenadler",
   "lng:de_DE:in_stock": true,
   "lng:de_DE:url": "https://www.shop.com/de_DE/products/desert-eagle",
   "lng:de_DE:image_url": "//images/de_DE/shoe.png",
   "lng:de_DE:display_price":"200€"
}]
CSV Product Feed Example 1
sku,group_id,name,in_stock,price,image_url,url,categories
111,22,t_shirt,true,9.50,http://dy.com/shop/111/ex.jpg,https://www.dy.com/shop/111.html?ch=22,mens|sport
CSV Product Feed Example 2
"sku","group_id","categories","name","url","price","image_url","in_stock","keywords","display_price","gender","product","surface","best_for_1","best_for_2","color_1","color_2","cushioning","support","lng:en_US:name","lng:en_US:in_stock","lng:en_US:url","lng:en_US:image_url","lng:en_US:display_price","lng:en_GB:name","lng:en_GB:in_stock","lng:en_GB:url","lng:en_GB:image_url","lng:en_GB:display_price","lng:fr_FR:name","lng:fr_FR:in_stock","lng:fr_FR:url","lng:fr_FR:image_url","lng:fr_FR:display_price","lng:de_DE:name","lng:de_DE:in_stock","lng:de_DE:url","lng:de_DE:image_url","lng:de_DE:display_price"
"1125-blue-42","1125","shoes|mens|road","Desert Eagle","https://www.shop.com/en_US/products/desert-eagle",225,"//images/en_US/shoe.png",true,"active|training|black|grey|direct|neutral","225$","mens","shoes","road","active","training","black","grey","direct","neutral","Desert Eagle",true,"https://www.shop.com/en_US/products/desert-eagle","//images/en_US/shoe.png","200$","Desert Eagle",false,"https://www.shop.com/en_GB/products/desert-eagle","//images/en_GB/shoe.png","200£","Aigle du désert",true,"https://www.shop.com/fr_FR/products/desert-eagle","//images/fr_FR/shoe.png","200€","Wüstenadler",true,"https://www.shop.com/de_DE/products/desert-eagle","//images/de_DE/shoe.png","200€

2. Sync the File with Dynamic Yield

If your feed file contains more than 20,000 items, contact your Customer Success Manager to sync your feed. If your feed is smaller, go to Assets › Data Feeds and create a new Product Feed.

Screen_Shot_2020-02-11_at_9.19.10.png

After naming the feed, you can choose one of the following sync options:

Sync a URL or upload a file: 

Upload a feed file (up to 25MB), or refer to a URL location where a feed file is stored. If you are using HTTP authentication, add the username and password to the feed URL as follows http://user:password@url.

You can click the "Add Another Source" to combine up to 10 sources: 

Screen_Shot_2020-02-13_at_21.52.37.png

You can then use a parser function to control how the two feed files are merged, as seen in the following example:

function parse(feed1, feed2) {
  var combinedFeeds =  feed1.concat(feed2);
  return combinedFeeds;
} 

Sync securely via Amazon S3

A password-protected S3 bucket managed by Dynamic Yield that will contain your product feed file.

  • Click Create a Bucket to create a new bucket. The credentials are only displayed once, so make sure you copy them to a secure location. You can generate new credentials using the Generate New Credentials option, but this is also limited to two times. Dynamic Yield will check the bucket for a new file six times a day. If you just uploaded a file to the bucket, you can also click - "sync now" to force the initial sync.
  • To perform the initial upload to the bucket, use the AWS CLI tool as seen in the example below for a US account (EU endpoints are different and provided in the admin upon feed creation). Note that the filename must be productfeed.csv.
  1. Push the file to S3:
    aws s3 cp local file .csv s3://com.dynamicyield.feeds/SECTION_ID/productfeed.csv
  2. Verify that the file is in S3: 
    aws s3 ls s3://com.dynamicyield.feeds/SECTION_ID/

    Make sure to add “/” at the end of the endpoint to avoid “access denied error” 

Screenshot_2020-05-17_at_16.14.34.png

Sync securely via SFTP

An S3 bucket secured using the SFTP protocol managed and provided by Dynamic Yield that will contain your product feed file.

  • Enter your SSH key to register it with Dynamic Yield (only one can be registered per Dynamic Yield site at any time).
  • Use the Endpoint, Server ID, and username to upload your feed (see screenshot below). The filename must be productfeed.csv. You can change the SSH key at any time using the Change Public SSH Key link.
  • Dynamic Yield will check the bucket for a new file six times a day. If you just uploaded a file to the bucket, you can also click - "sync now" to force the initial sync.

image__2_.png

Frequency

By default, the product feed will sync with the source file every 24 hours. However, you can change it if your product catalog updates more frequently. This is under the Advanced Settings. If you sync the file via S3 / SFTP - Dynamic Yield will check the bucket for a new file six times a day. If you just uploaded a file to the bucket, you can also click - "sync now" to force the initial sync.

Alternatively to periodic sync, you can use an API to update the feed on demand. This option lets you add, delete, or modify entire rows or values in your data feed and sync it faster to Dynamic Yield. However, it requires advanced setup and technical support. Contact your Customer Success Manager for more details.  

Screen_Shot_2020-02-11_at_9.27.50.png

3. Parse the Source File (Optional)

If the columns and values in your feed file don’t meet the requirements above, or if some other manipulation is required in order to duplicate, move around or rename values and fields, you can modify the feed using a parser function created in JavaScript. For example, XML files which are not "flat" can be "flattened" using a parser function. 

Note: Parser functions are only supported for data feeds up to 200,000 items and 180MB. 

To add a parser function, go to Advanced Settings › Parser Function and enter your code.

Example - Parse an XML Feed 

Parser functions are required for XML feeds so the feed can be converted into an array inside a JSON.

function parser(data){
 var products = data.items[0].item;
 return products.map(function(item){
 var newFeedRow = {};
 for(var column in item){
 newFeedRow[column] = item[column][0];
 }
 return newFeedRow;
 });
}
Example - Combine Two Feeds 

You can use a parser function to combine two product feeds into one feed.

function parse(feed1, feed2) {
  var combinedFeeds =  feed1.concat(feed2);
  return combinedFeeds;
} 
Example - Import Google Feed File

GPF (Google Product Feed), commonly used for Shopping Ads, is very similar to the Dynamic Yield’s Product Feed structure. If you already have a GPF, you can use the following parser function code as an example, and adjust it to fit to your GPF.

function parse(data) {
 const DY_GOOGLE_FIELD_MAPPING = {
 'sku': 'g:id',
 'name': 'g:title',
 'url': 'g:link',
 'price': 'g:price',
 'in_stock': 'g:availability',
 'image_url': 'g:image_link',
 'categories': 'g:product_type',
 'group_id': 'g:item_group_id'
 };
 /*
 in case the category is constructed from several feeds with the same name and a number,
 for example: "g:category_level1","g:category_level2","g:category_level3":
 */
 const DY_GOOGLE_FIELD_MAPPING_CATEGORIES_FALLBACK = 'g:category_level*';
 const MAX_FALLBACK_CATEGORIES = 10;
 const TRY_FINDING_ITEMS_IN_XML_AUTOMATICALLY = true;
 const ONLY_ADD_ITEMS_FROM_FIELD_MAPPING = false;
 /* In case the automatic item allocation is not used:*/
 const FEED_ITEMS_ARRAY_PATH = data.rss[0].channel[0].item;

 const parseItem = oldItem => {
 let newItem = {};
 for (let dyFieldName in DY_GOOGLE_FIELD_MAPPING) {
 if (!DY_GOOGLE_FIELD_MAPPING.hasOwnProperty(dyFieldName)) {
 continue;
 }
 switch (true) {
 case dyFieldName === 'sku':
 case dyFieldName === 'name':
 case dyFieldName === 'url':
 case dyFieldName === 'image_url':
 case (dyFieldName === 'group_id' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.group_id] !== 'undefined'):
 newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]);
 break;
 /* if no group id found in the original feed, use the sku instead:*/ 
 case (dyFieldName === 'group_id' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.group_id] === 'undefined'):
 newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING.sku);
 break;
 case dyFieldName === 'in_stock':
 newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]) === 'in stock';
 break;
 case dyFieldName === 'price':
 newItem[dyFieldName] = parseFloat(stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]).replace(',', '.').replace(/[^0-9\.-]+/g, ''));
 break;
 case (dyFieldName === 'categories' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.categories] !== 'undefined'):
 newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]).replace(/\ >\ /g, '|');
 break;
 /* if the category feed doesn't exist, loop through the fallback value*/
 case (dyFieldName === 'categories' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.categories] === 'undefined'):
 let categoryStr = '';
 for (let i = 0; i < MAX_FALLBACK_CATEGORIES; i++) { let categoryFieldName = DY_GOOGLE_FIELD_MAPPING_CATEGORIES_FALLBACK.replace('*', i); if (oldItem[categoryFieldName] && oldItem[categoryFieldName][0]) { if (categoryStr.length) { categoryStr += '|'; } categoryStr += oldItem[categoryFieldName][0]; // delete oldItem[categoryFieldName]; } if (categoryStr.length && typeof oldItem[categoryFieldName] === 'undefined') { break; } } newItem[dyFieldName] = categoryStr; break; default: } } if (!ONLY_ADD_ITEMS_FROM_FIELD_MAPPING) { for (let customerField in oldItem) { if (!oldItem.hasOwnProperty(customerField)) { continue; } let fieldValue = stringifyValue(oldItem, customerField); if (typeof fieldValue === 'object') { // this currently support nested objects for non mandatory/mapped fields. parseNestedObject(newItem, oldItem, customerField); } else { newItem[customerField] = fieldValue; } } } return newItem; }; /* this funtion scans the xml for array larger than 1, which will most likley only be the array of items. */ const getItemsNode = feed => {
 let elements = [feed];
 for (let maxTries = 100; maxTries > 0; maxTries--) {
 if (!elements.length) {
 break;
 }
 let element = elements.pop();
 if (typeof element !== 'object') {
 continue;
 }
 if (element.length && element.length > 1) {
 return element;
 }
 if (element.length) {
 elements.push(element[0]);
 } else {
 for (let child in element) {
 if (!element.hasOwnProperty(child)) {
 continue;
 }
 elements.push(element[child]);
 }
 }
 }
 return true;
 };

 /* tries to return the value as a string, else return the value as is*/
 const stringifyValue = (oldItem, field) => {
 if (!oldItem[field]) {
 return '';
 }
 let fieldValue = oldItem[field][0];
 if (typeof fieldValue !== 'object') {
 return (fieldValue + '').replace('<![CDATA[', '').replace(']]>', '');
 }
 return fieldValue;
 };

 /*
 some google fields contains nested objects. This only covers one child hirarchy and
 naming the nested field in this patern: parent_child_i for example: 'g:shipping_g:country_1'
 */
 const parseNestedObject = (newItem, oldItem, parentFieldName) => {
 for (let i = 0; i < oldItem[parentFieldName].length; i++) {
 let nestedFieldObject = oldItem[parentFieldName][i];
 for (let nestedField in nestedFieldObject) {
 if (!nestedFieldObject.hasOwnProperty(nestedField)) {
 continue;
 }
 newItem[parentFieldName + '_' + nestedField + '_' + i] = nestedFieldObject[nestedField][0];
 }
 }
 };


 let oldItems;
 if (TRY_FINDING_ITEMS_IN_XML_AUTOMATICALLY) {
 oldItems = getItemsNode(data);
 } else {
 oldItems = FEED_ITEMS_ARRAY_PATH;
 }

 return oldItems.map(parseItem);
}

 4. Save and Preview

If your feed source is an uploaded file or URL, you can view a preview of up to 100 entries of your feed including any errors or warning by clicking Preview. Click Save and Activate to proceed or Cancel to go back. 

feed_preview.png

The synchronization duration depends on the number of items in your feed. For feeds with 250,000 items in the feed it usually takes a few minutes, and 30 minutes for every additional 500,000 items in your feed. If you require a faster sync duration, consider updating your feed with API to update incremental changes instead of the standard full-feed update. 

5. Validate Sync Status

You can verify that your feed is synchronizing successfully at any time.  

  1. Go to Assets › Data Feeds.
  2. In the Last Sync Attempt column, you should see a Date Feed Synced icon sync_icon.png and a recent date. If the icon indicates an error or warning, download the log for more details.
  3. In the Actions column, click the View icon View_icon.png. Validate that all required columns and any custom columns you have added exist and have the correct data. 

Note: For feeds over 2M product there will be products filtered from the data feeds preview that there are no events for. Active products are products that have one or more activities (product view / add-to-cart / purchase) in the last 6 months. So when you viewing the product feed in the preview, the number of products at the bottom of the page is the updated number once applying this filter.