A Product Feed is a file that contains your product catalog (all your products, with product metadata). It powers affinity-based personalization, product recommendations, social proof messaging, and more.
You can create product feeds by syncing data feed files as specified in this article or create and sync a product feed using APIs.
Creating a Product Feed Overview
Syncing your product catalog with Dynamic Yield includes the following steps:
- Create the file with all the required product metadata (e.g. price, SKU). This is done on your end.
- Sync the file with Dynamic Yield by referring to a URL, uploading it to a secured S3 bucket directly, or uploading it to S3 via SFTP. To sync the feed using APIs, see Creating and Sync a Product Feed using APIs.
- Parse the source file, if needed, by writing a JavaScript code that manipulates the source file to adjust it to the format required by Dynamic Yield (e.g. changing attribute names to match to Dynamic Yield requirements). This is done in the Product Feed advanced settings, and is only supported for feeds that are smaller than 200,000 items or 180MB.
- Save and preview to check if the feed meets all Dynamic Yield requirements. Afterwards you can save and publish.
Note: If your feed is larger than 20,000 items, please contact your Customer Success Manager. The syncing process (Steps 2-4) will be done by them. The total feed size is limited to 5 Milion items.
1. Create the File
Before you can upload and sync a Product Feed to Dynamic Yield, you need to make sure your feed file complies with size, format, and content requirements.
File Type and Size
Product Feeds can be in CSV, JSON, or XML format. However, if the feed contains more than 200,000 items or is larger than 180MB, it must be in CSV format. It can be stored in any URL or a local folder, or in an Amazon S3 bucket that we provide. You can transfer the feeds to Amazon S3 using SFTP as well.
If you use a CSV file, items must be separated by commas.
Product Feed Structure
The Product Feed must include the basic information about your product (SKU, group ID, price, image, URL, if it's in stock or not, and its categories). You can see the list of mandatory attributes below, with detailed instructions (format, limitations, etc.):
| Column (case sensitive) | Description | Format | Example |
|---|---|---|---|
| sku | Product Unique Identifier. Note: SKUs cannot contain spaces or the string "//" | String (up to 256 characters, no spaces) |
111 |
| group_id | Identifies a group of products that differ in some product attributes. Note: this value cannot contain spaces. | String (up to 256 characters, no spaces) |
22 |
| name | Product Name | String (up to 1,000 characters) |
t_shirt |
| url | URL to the product details page (must be a valid URL, starting with HTTP/HTTPS) | String (up to 1,000 characters) |
http://mystore.com/111 |
| price | Price of the product (must be a number) | Float (up to 1,000 characters) |
9.50 |
| in_stock | Indication whether the product is in stock (noted by ‘true/false’) | Boolean (up to 1,000 characters) |
true |
| image_url | The URL to the product image (must be a valid URL, starting with HTTP/HTTPS) | String (up to 1,000 characters) |
http://mystore.com/111.png |
| categories | The categories associated with the product from general to specific | Strings (separated by pipes with no spaces, up to 1,000 characters) |
Women’s|Trousers & Jeans|Chinos |
| keywords (optional but strongly recommended) | Any additional information describing the product, separated by pipes. Used for our machine learning and affinity algorithms. Because this column is optional, blank values will not trigger errors or warnings. | String (up to 1,000 characters) |
sale_item |
All of the columns listed above, except "keywords", are mandatory. If a product is missing a value in one of these columns, it will be skipped. If more than 10% of products have an empty value in one of the columns, the entire sync will fail.
Note: If your feed source has this information, but names are different (for example, if you have a Google Product Feed), or this information exists in 2 separate feeds - you can use a parser function when syncing the feed to adapt it according to the guidelines above. This is available for feeds that have less than 200,000 items and 180MB.
Custom Columns
You can also add more columns to the feed that describe your product. These can later be used for targeting (e.g. if you add "color", you can use it to target users who purchased blue items), affinity score (e.g. include color affinity in the affinity recommendation algorithm), define merchandising rules in your recommendation widgets (e.g. never recommend products with a specific brand).
Guidelines:
- Custom columns values must be string, up to 1,000 characters, with no special characters.
- Custom column names must not start with '_id'.
- Once you add a column, you cannot remove it. You can stop using it or return empty values, but the column must be in any future sync.
*You can create a total of up to 300 columns when working with multiple languages (including all mandatory, custom, and translated columns).
Multi-Language Support
To support multiple languages, you can specify different values for different languages for any field. Use the format “lng:<language code>:<column name>”: “<value>”. For example, if wanted to have a translation for a product name, you would create two rows for the product as follows:
"lng:en_EN:name":"White Pants"
"lng:de_DE:name":"weiße Hosen"
The language code should match the name you use in the page context. For details, see Page Context.
Because these columns are optional, blank values will not trigger errors or warnings.
*You can create a total of up to 300 columns when working with multiple languages (including all mandatory, custom, and translated columns).
Note: While Dynamic Yield supports feeds of up to 5 million items, there are some limitations as the size of the feed increases. Additionally, larger feeds will experience slower sync times as well as slower recommendation serving speed.
Product Feed File Examples
<items>
<item>
<sku>111</sku>
<group_id>22</group_id>
<price>9.5</price>
<categories>mens|sport</categories>
<in_stock>true</in_stock>
<name>t_shirt</name>
<image_url>http://dy.com/shop/111/ex.jpg</image_url>
<url>https://www.dy.com/shop/111.html?ch=22</url>
</item>
</items> <items>
<item>
<sku>1125-blue-42</sku>
<group_id>1125</group_id>
<categories>shoes|mens|road</categories>
<name>Desert Eagle</name>
<url>https://www.shop.com/en_US/products/desert-eagle</url>
<price>225</price>
<image_url>//images/en_US/shoe.png</image_url>
<in_stock>true</in_stock>
<keywords>active|training|black|grey|direct|neutral</keywords>
<display_price>225$</display_price>
<gender>mens</gender>
<product>shoes</product>
<surface>road</surface>
<best_for_1>active</best_for_1>
<best_for_2>training</best_for_2>
<color_1>black</color_1>
<color_2>grey</color_2>
<cushioning>direct</cushioning>
<support>neutral</support>
<lng:en_US:name>Desert Eagle</lng:en_US:name>
<lng:en_US:in_stock>true</lng:en_US:in_stock>
<lng:en_US:url>https://www.shop.com/en_US/products/desert-eagle</lng:en_US:url>
<lng:en_US:image_url>//images/en_US/shoe.png</lng:en_US:image_url>
<lng:en_US:display_price>200$</lng:en_US:display_price>
<lng:en_GB:name>Desert Eagle</lng:en_GB:name>
<lng:en_GB:in_stock>false</lng:en_GB:in_stock>
<lng:en_GB:url>https://www.shop.com/en_GB/products/desert-eagle</lng:en_GB:url>
<lng:en_GB:image_url>//images/en_GB/shoe.png</lng:en_GB:image_url>
<lng:en_GB:display_price>200£</lng:en_GB:display_price>
<lng:fr_FR:name>Aigle du désert</lng:fr_FR:name>
<lng:fr_FR:in_stock>true</lng:fr_FR:in_stock>
<lng:fr_FR:url>https://www.shop.com/fr_FR/products/desert-eagle</lng:fr_FR:url>
<lng:fr_FR:image_url>//images/fr_FR/shoe.png</lng:fr_FR:image_url>
<lng:fr_FR:display_price>200€</lng:fr_FR:display_price>
<lng:de_DE:name>Wüstenadler</lng:de_DE:name>
<lng:de_DE:in_stock>true</lng:de_DE:in_stock>
<lng:de_DE:url>https://www.shop.com/de_DE/products/desert-eagle</lng:de_DE:url>
<lng:de_DE:image_url>//images/de_DE/shoe.png</lng:de_DE:image_url>
<lng:de_DE:display_price>200€</lng:de_DE:display_price>
</item>
</items>
[{
"sku": "111",
"group_id": "22",
"price": 9.50,
"name": "t_shirt",
"in_stock": true,
"image_url": "http://dy.com/shop/111/ex.jpg",
"url": "https://www.dy.com/shop/111.html?ch=22",
"categories": "mens|sport"
}
{
"sku": "123",
"group_id": "22",
"price": 8.50,
"name": "t_shirt",
"in_stock": true,
"image_url": "http://dy.com/shop/123/ex.jpg",
"url": "https://www.dy.com/shop/123.html?ch=22",
"categories": "mens|sport"
}
]
[{
"sku": "1125-blue-42",
"group_id": "1125",
"categories": "shoes|mens|road",
"name": "Desert Eagle",
"url": "https://www.shop.com/en_US/products/desert-eagle",
"price": 225.0,
"image_url": "//images/en_US/shoe.png",
"in_stock": true,
"keywords": "active|training|black|grey|direct|neutral",
"display_price":"225$",
"gender": "mens",
"product": "shoes",
"surface": "road",
"best_for_1": "active",
"best_for_2": "training",
"color_1": "black",
"color_2": "grey",
"cushioning": "direct",
"support": "neutral",
"lng:en_US:name":"Desert Eagle",
"lng:en_US:in_stock": true,
"lng:en_US:url": "https://www.shop.com/en_US/products/desert-eagle",
"lng:en_US:image_url": "//images/en_US/shoe.png",
"lng:en_US:display_price":"200$",
"lng:en_GB:name":"Desert Eagle",
"lng:en_GB:in_stock": false,
"lng:en_GB:url": "https://www.shop.com/en_GB/products/desert-eagle",
"lng:en_GB:image_url": "//images/en_GB/shoe.png",
"lng:en_GB:display_price":"200£",
"lng:fr_FR:name":"Aigle du désert",
"lng:fr_FR:in_stock": true,
"lng:fr_FR:url": "https://www.shop.com/fr_FR/products/desert-eagle",
"lng:fr_FR:image_url": "//images/fr_FR/shoe.png",
"lng:fr_FR:display_price":"200€",
"lng:de_DE:name":"Wüstenadler",
"lng:de_DE:in_stock": true,
"lng:de_DE:url": "https://www.shop.com/de_DE/products/desert-eagle",
"lng:de_DE:image_url": "//images/de_DE/shoe.png",
"lng:de_DE:display_price":"200€"
}]
sku,group_id,name,in_stock,price,image_url,url,categories
111,22,t_shirt,true,9.50,http://dy.com/shop/111/ex.jpg,https://www.dy.com/shop/111.html?ch=22,mens|sport
"sku","group_id","categories","name","url","price","image_url","in_stock","keywords","display_price","gender","product","surface","best_for_1","best_for_2","color_1","color_2","cushioning","support","lng:en_US:name","lng:en_US:in_stock","lng:en_US:url","lng:en_US:image_url","lng:en_US:display_price","lng:en_GB:name","lng:en_GB:in_stock","lng:en_GB:url","lng:en_GB:image_url","lng:en_GB:display_price","lng:fr_FR:name","lng:fr_FR:in_stock","lng:fr_FR:url","lng:fr_FR:image_url","lng:fr_FR:display_price","lng:de_DE:name","lng:de_DE:in_stock","lng:de_DE:url","lng:de_DE:image_url","lng:de_DE:display_price"
"1125-blue-42","1125","shoes|mens|road","Desert Eagle","https://www.shop.com/en_US/products/desert-eagle",225,"//images/en_US/shoe.png",true,"active|training|black|grey|direct|neutral","225$","mens","shoes","road","active","training","black","grey","direct","neutral","Desert Eagle",true,"https://www.shop.com/en_US/products/desert-eagle","//images/en_US/shoe.png","200$","Desert Eagle",false,"https://www.shop.com/en_GB/products/desert-eagle","//images/en_GB/shoe.png","200£","Aigle du désert",true,"https://www.shop.com/fr_FR/products/desert-eagle","//images/fr_FR/shoe.png","200€","Wüstenadler",true,"https://www.shop.com/de_DE/products/desert-eagle","//images/de_DE/shoe.png","200€
2. Sync the File with Dynamic Yield
If your feed file contains more than 20,000 items, contact your Customer Success Manager to sync your feed. If your feed is smaller, go to Assets › Data Feeds and create a new Product Feed.
After naming the feed, you can choose one of the following sync options:
Sync a URL or upload a file:
Upload a feed file (up to 25MB), or refer to a URL location where a feed file is stored. If you are using HTTP authentication, add the username and password to the feed URL as follows http://user:password@url.
You can click the "Add Another Source" to combine up to 10 sources:
You can then use a parser function to control how the two feed files are merged, as seen in the following example:
function parse(feed1, feed2) {
var combinedFeeds = feed1.concat(feed2);
return combinedFeeds;
} Sync securely via Amazon S3
A password-protected S3 bucket managed by Dynamic Yield that will contain your product feed file. Click Create a Bucket to create a new bucket. The credentials are only displayed once, so make sure you copy them to a secure location. You can generate new credentials using the Generate New Credentials option, but this is also limited to two times. Dynamic Yield will check the bucket for a new file six times a day. If you just uploaded a file to the bucket, you can also click - "sync now" to force the initial sync.
To perform the initial upload to the bucket, use the AWS CLI tool as seen in the example below for a US account (EU endpoints are different and provided in the admin upon feed creation). Note that the filename must be productfeed.csv.
- Push the file to S3:
aws s3 cp local file .csv s3://com.dynamicyield.feeds/SECTION_ID/productfeed.csv - Verify that the file is in S3:
aws s3 ls s3://com.dynamicyield.feeds/SECTION_ID/
Make sure to add “/” at the end of the endpoint to avoid “access denied error”
Sync securely via SFTP
An S3 bucket secured using the SFTP protocol managed by Dynamic Yield that will contain your product feed file. Enter your SSH key to register it with Dynamic Yield (only one can be registered per Dynamic Yield site at any time). Use the Endpoint, Server ID, and username to upload your feed. The filename must be productfeed.csv. You can change the SSH key at any time using the Change Public SSH Key. Dynamic Yield will check the bucket for a new file six times a day. If you just uploaded a file to the bucket, you can also click - "sync now" to force the initial sync.
Frequency
By default, the product feed will sync with the source file every 24 hours. However, you can change it if your product catalog updates more frequently. This is under the Advanced Settings. If you sync the file via S3 / SFTP - Dynamic Yield will check the bucket for a new file six times a day. If you just uploaded a file to the bucket, you can also click - "sync now" to force the initial sync.
Alternatively to periodic sync, you can use an API to update the feed on demand. This option lets you add, delete, or modify entire rows or values in your data feed and sync it faster to Dynamic Yield. However, it requires advanced setup and technical support. Contact your Customer Success Manager for more details.
3. Parse the Source File (Optional)
If the columns and values in your feed file don’t meet the requirements above, or if some other manipulation is required in order to duplicate, move around or rename values and fields, you can modify the feed using a parser function created in JavaScript. For example, XML files which are not "flat" can be "flattened" using a parser function.
Note: Parser functions are only supported for data feeds up to 200,000 items and 180MB.
To add a parser function, go to Advanced Settings › Parser Function and enter your code.
Parser functions are required for XML feeds so the feed can be converted into an array inside a JSON.
function parser(data){
var products = data.items[0].item;
return products.map(function(item){
var newFeedRow = {};
for(var column in item){
newFeedRow[column] = item[column][0];
}
return newFeedRow;
});
}You can use a parser function to combine two product feeds into one feed.
function parse(feed1, feed2) {
var combinedFeeds = feed1.concat(feed2);
return combinedFeeds;
} GPF (Google Product Feed), commonly used for Shopping Ads, is very similar to the Dynamic Yield’s Product Feed structure. If you already have a GPF, you can use the following parser function code as an example, and adjust it to fit to your GPF.
function parse(data) {
const DY_GOOGLE_FIELD_MAPPING = {
'sku': 'g:id',
'name': 'g:title',
'url': 'g:link',
'price': 'g:price',
'in_stock': 'g:availability',
'image_url': 'g:image_link',
'categories': 'g:product_type',
'group_id': 'g:item_group_id'
};
/*
in case the category is constructed from several feeds with the same name and a number,
for example: "g:category_level1","g:category_level2","g:category_level3":
*/
const DY_GOOGLE_FIELD_MAPPING_CATEGORIES_FALLBACK = 'g:category_level*';
const MAX_FALLBACK_CATEGORIES = 10;
const TRY_FINDING_ITEMS_IN_XML_AUTOMATICALLY = true;
const ONLY_ADD_ITEMS_FROM_FIELD_MAPPING = false;
/* In case the automatic item allocation is not used:*/
const FEED_ITEMS_ARRAY_PATH = data.rss[0].channel[0].item;
const parseItem = oldItem => {
let newItem = {};
for (let dyFieldName in DY_GOOGLE_FIELD_MAPPING) {
if (!DY_GOOGLE_FIELD_MAPPING.hasOwnProperty(dyFieldName)) {
continue;
}
switch (true) {
case dyFieldName === 'sku':
case dyFieldName === 'name':
case dyFieldName === 'url':
case dyFieldName === 'image_url':
case (dyFieldName === 'group_id' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.group_id] !== 'undefined'):
newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]);
break;
/* if no group id found in the original feed, use the sku instead:*/
case (dyFieldName === 'group_id' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.group_id] === 'undefined'):
newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING.sku);
break;
case dyFieldName === 'in_stock':
newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]) === 'in stock';
break;
case dyFieldName === 'price':
newItem[dyFieldName] = parseFloat(stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]).replace(',', '.').replace(/[^0-9\.-]+/g, ''));
break;
case (dyFieldName === 'categories' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.categories] !== 'undefined'):
newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]).replace(/\ >\ /g, '|');
break;
/* if the category feed doesn't exist, loop through the fallback value*/
case (dyFieldName === 'categories' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.categories] === 'undefined'):
let categoryStr = '';
for (let i = 0; i < MAX_FALLBACK_CATEGORIES; i++) { let categoryFieldName = DY_GOOGLE_FIELD_MAPPING_CATEGORIES_FALLBACK.replace('*', i); if (oldItem[categoryFieldName] && oldItem[categoryFieldName][0]) { if (categoryStr.length) { categoryStr += '|'; } categoryStr += oldItem[categoryFieldName][0]; // delete oldItem[categoryFieldName]; } if (categoryStr.length && typeof oldItem[categoryFieldName] === 'undefined') { break; } } newItem[dyFieldName] = categoryStr; break; default: } } if (!ONLY_ADD_ITEMS_FROM_FIELD_MAPPING) { for (let customerField in oldItem) { if (!oldItem.hasOwnProperty(customerField)) { continue; } let fieldValue = stringifyValue(oldItem, customerField); if (typeof fieldValue === 'object') { // this currently support nested objects for non mandatory/mapped fields. parseNestedObject(newItem, oldItem, customerField); } else { newItem[customerField] = fieldValue; } } } return newItem; }; /* this funtion scans the xml for array larger than 1, which will most likley only be the array of items. */ const getItemsNode = feed => {
let elements = [feed];
for (let maxTries = 100; maxTries > 0; maxTries--) {
if (!elements.length) {
break;
}
let element = elements.pop();
if (typeof element !== 'object') {
continue;
}
if (element.length && element.length > 1) {
return element;
}
if (element.length) {
elements.push(element[0]);
} else {
for (let child in element) {
if (!element.hasOwnProperty(child)) {
continue;
}
elements.push(element[child]);
}
}
}
return true;
};
/* tries to return the value as a string, else return the value as is*/
const stringifyValue = (oldItem, field) => {
if (!oldItem[field]) {
return '';
}
let fieldValue = oldItem[field][0];
if (typeof fieldValue !== 'object') {
return (fieldValue + '').replace('<![CDATA[', '').replace(']]>', '');
}
return fieldValue;
};
/*
some google fields contains nested objects. This only covers one child hirarchy and
naming the nested field in this patern: parent_child_i for example: 'g:shipping_g:country_1'
*/
const parseNestedObject = (newItem, oldItem, parentFieldName) => {
for (let i = 0; i < oldItem[parentFieldName].length; i++) {
let nestedFieldObject = oldItem[parentFieldName][i];
for (let nestedField in nestedFieldObject) {
if (!nestedFieldObject.hasOwnProperty(nestedField)) {
continue;
}
newItem[parentFieldName + '_' + nestedField + '_' + i] = nestedFieldObject[nestedField][0];
}
}
};
let oldItems;
if (TRY_FINDING_ITEMS_IN_XML_AUTOMATICALLY) {
oldItems = getItemsNode(data);
} else {
oldItems = FEED_ITEMS_ARRAY_PATH;
}
return oldItems.map(parseItem);
}4. Save and Preview
If your feed source is an uploaded file or URL, you can view a preview of up to 100 entries of your feed including any errors or warning by clicking Preview. Click Save and Activate to proceed or Cancel to go back.
The synchronization duration depends on the number of items in your feed. For feeds with 250,000 items in the feed it usually takes a few minutes, and 30 minutes for every additional 500,000 items in your feed. If you require a faster sync duration, consider updating your feed with API to update incremental changes instead of the standard full-feed update.
5. Validate Sync Status
You can verify that your feed is synchronizing successfully at any time.
- Go to Assets › Data Feeds.
- In the Last Sync Attempt column, you should see a Date Feed Synced icon
and a recent date. If the icon indicates an error or warning, download the log for more details. - In the Actions column, click the View icon
. Validate that all required columns and any custom columns you have added exist and have the correct data.