A product feed is a file that contains your product catalog (all your products, with product metadata). It powers affinity-based personalization, product recommendations, social proof messaging, and more.
You can create product feeds by syncing data feed files as specified in this article or you can create and sync your product feed using APIs.
Creating a product feed: Overview
Syncing your product catalog with Experience OS includes the following steps (click the numbered links to go to that section of this article):
- Create the file with all the required product metadata (price, SKU). This is done on your end.
- Sync the file with Experience OS by referring to a URL, uploading it to a secured S3 bucket directly, or uploading it to S3 via SFTP. To sync the feed using APIs, see Creating and Syncing a Product Feed using APIs.
- Parse the source file, if needed, by writing JavaScript code that manipulates the source file to adjust it to the format required by Experience OS (such as changing attribute names). This is done in the product feed advanced settings, and is supported only for feeds that are smaller than 200,000 items or 180 MB.
- Save and preview to check whether the feed meets all Experience OS requirements. Then, you can save and publish.
Note: If your feed includes more than 20,000 items, contact your Technical Account Manager. They will do the sync process (steps 2-4). The total feed size is limited to 5 million items.
Check out our best practices for product feed design to get the most our of your product feed
Step 1: Create the file
Before you can upload and sync a product feed to Experience OS, make sure your feed file complies with the size, format, and content requirements.
File type and size
| Feed Size | File Type |
| Up to 20,000 items | CSV, JSON, XML |
| 20,000-200,000 | CSV, JSON |
| More than 200,000 | CSV |
The file can be stored in any URL or a local folder, or in an Amazon S3 bucket that we provide. You can transfer the feeds to Amazon S3 using SFTP as well.
If you use a CSV file, items must be separated by commas.
Product feed structure
The product feed must include the basic information about your product (SKU, group ID, price, image, URL, if it's in stock or not, and its categories). You can see the list of mandatory attributes in the following table, with detailed instructions (format, limitations, and so on).
SKUs must be unique, or the feed won't sync. For other mandatory fields, as long as duplications do not exceed 10% of total values, the feed will synch.
Note: If you change the group ID structure or category structure, the feed might reset, causing all previous recommendation data to be lost.
| Column (case sensitive) | Description | Format | Example |
|---|---|---|---|
| sku | Product Unique Identifier. Note: SKUs cannot contain spaces or the string "//" | String (up to 256 characters, no spaces) |
111 |
| group_id |
Identifies a group of products that differ in some product attributes. |
String (up to 256 characters, no spaces) |
22 |
| name | Product Name | String (up to 1,000 characters) |
t_shirt |
| url | URL to the product details page (must be a valid URL, starting with HTTP/HTTPS) | String (up to 1,000 characters) |
http://mystore.com/111 |
| price | Price of the product (must be a number) | Float (up to 1,000 characters) |
9.50 |
| in_stock | Indication whether the product is in stock (noted by ‘true/false’) | Boolean (must be lower case) |
true |
| image_url | The URL to the product image (must be a valid URL, starting with HTTP/HTTPS) | String (up to 1,000 characters) |
http://mystore.com/111.png |
| categories | The categories associated with the product from general to specific | Strings (separated by pipes with no spaces, up to 1,000 characters) |
Women’s|Trousers & Jeans|Chinos |
| keywords (optional but strongly recommended) | Any additional information describing the product, separated by pipes. Used for our machine learning and affinity algorithms. Because this column is optional, blank values will not trigger errors or warnings. | String (up to 1,000 characters) |
sale_item |
All of the columns listed above, except "keywords", are mandatory. If a product is missing a value in one of these columns, it will be skipped. If more than 10% of products have an empty value in one of the columns, the entire sync will fail.
Note: If your feed source has this information, but names are different (for example, if you have a Google Product Feed), or this information exists in 2 separate feeds, you can use a parser function when syncing the feed to adapt it according to the guidelines above. This is available for feeds that have fewer than 200,000 items and are smaller than 180 MB.
Custom columns
You can also add more columns to the feed that describe your product. These can later be used for targeting (for example, if you add "color", you can use it to target users who purchased blue items), affinity score (say, include color affinity in the affinity recommendation algorithm), define merchandising rules in your recommendation widgets (perhaps, never recommend products with a specific brand).
Guidelines:
- Custom column values must be strings, up to 1,000 characters, with no special characters. Special characters are permitted as long as they are wrapped within a string.
- Custom column names must not start with "id".
- After you add a column, you can't remove it. You can stop using it or return empty values, but the column must be included in any future sync.
*You can create a total of up to 300 columns when working with multiple languages (including all mandatory, custom, and translated columns).
Multi-language support
To support multiple languages, you can specify different values for different languages for any field. Use the format “lng:<language code>:<column name>”: “<value>”. For example, if wanted to have a translation for a product name, you would create two rows for the product as follows:
"lng:en_EN:name":"White Pants"
"lng:de_DE:name":"weiße Hosen"
The language code should match the name you use in the page context. For details, see Page Context. The language code can have up to two (_) or (-) between the values. For example: "lng:en_EN_X".
Because these columns are optional, blank values do not trigger errors or warnings.
You can create a total of up to 300 columns when working with multiple languages (including all mandatory, custom, and translated columns).
To learn how to target campaigns based on user locale (using the "lng" attribute), see the Multi-language Support article.
Notes:
- Feed size: While Experience OS supports feeds of up to 5 million items, there are some limitations as the size of the feed increases. Additionally, larger feeds will experience slower sync times as well as slower recommendation serving speed.
- Translated content: Multi-language sites cannot use translated content in context or feed to match the set default language ("lng").
Product feed file examples
<items>
<item>
<sku>111</sku>
<group_id>22</group_id>
<price>9.5</price>
<categories>mens|sport</categories>
<in_stock>true</in_stock>
<name>t_shirt</name>
<image_url>http://dy.com/shop/111/ex.jpg</image_url>
<url>https://www.dy.com/shop/111.html?ch=22</url>
</item>
</items> <items>
<item>
<sku>1125-blue-42</sku>
<group_id>1125</group_id>
<categories>shoes|mens|road</categories>
<name>Desert Eagle</name>
<url>https://www.shop.com/en_US/products/desert-eagle</url>
<price>225</price>
<image_url>//images/en_US/shoe.png</image_url>
<in_stock>true</in_stock>
<keywords>active|training|black|grey|direct|neutral</keywords>
<display_price>225$</display_price>
<gender>mens</gender>
<product>shoes</product>
<surface>road</surface>
<best_for_1>active</best_for_1>
<best_for_2>training</best_for_2>
<color_1>black</color_1>
<color_2>grey</color_2>
<cushioning>direct</cushioning>
<support>neutral</support>
<lng:en_US:name>Desert Eagle</lng:en_US:name>
<lng:en_US:in_stock>true</lng:en_US:in_stock>
<lng:en_US:url>https://www.shop.com/en_US/products/desert-eagle</lng:en_US:url>
<lng:en_US:image_url>//images/en_US/shoe.png</lng:en_US:image_url>
<lng:en_US:display_price>200$</lng:en_US:display_price>
<lng:en_GB:name>Desert Eagle</lng:en_GB:name>
<lng:en_GB:in_stock>false</lng:en_GB:in_stock>
<lng:en_GB:url>https://www.shop.com/en_GB/products/desert-eagle</lng:en_GB:url>
<lng:en_GB:image_url>//images/en_GB/shoe.png</lng:en_GB:image_url>
<lng:en_GB:display_price>200£</lng:en_GB:display_price>
<lng:fr_FR:name>Aigle du désert</lng:fr_FR:name>
<lng:fr_FR:in_stock>true</lng:fr_FR:in_stock>
<lng:fr_FR:url>https://www.shop.com/fr_FR/products/desert-eagle</lng:fr_FR:url>
<lng:fr_FR:image_url>//images/fr_FR/shoe.png</lng:fr_FR:image_url>
<lng:fr_FR:display_price>200€</lng:fr_FR:display_price>
<lng:de_DE:name>Wüstenadler</lng:de_DE:name>
<lng:de_DE:in_stock>true</lng:de_DE:in_stock>
<lng:de_DE:url>https://www.shop.com/de_DE/products/desert-eagle</lng:de_DE:url>
<lng:de_DE:image_url>//images/de_DE/shoe.png</lng:de_DE:image_url>
<lng:de_DE:display_price>200€</lng:de_DE:display_price>
</item>
</items>
[{
"sku": "111",
"group_id": "22",
"price": 9.50,
"name": "t_shirt",
"in_stock": true,
"image_url": "http://dy.com/shop/111/ex.jpg",
"url": "https://www.dy.com/shop/111.html?ch=22",
"categories": "mens|sport"
},
{
"sku": "123",
"group_id": "22",
"price": 8.50,
"name": "t_shirt",
"in_stock": true,
"image_url": "http://dy.com/shop/123/ex.jpg",
"url": "https://www.dy.com/shop/123.html?ch=22",
"categories": "mens|sport"
}
]
[{
"sku": "1125-blue-42",
"group_id": "1125",
"categories": "shoes|mens|road",
"name": "Desert Eagle",
"url": "https://www.shop.com/en_US/products/desert-eagle",
"price": 225.0,
"image_url": "//images/en_US/shoe.png",
"in_stock": true,
"keywords": "active|training|black|grey|direct|neutral",
"display_price":"225$",
"gender": "mens",
"product": "shoes",
"surface": "road",
"best_for_1": "active",
"best_for_2": "training",
"color_1": "black",
"color_2": "grey",
"cushioning": "direct",
"support": "neutral",
"lng:en_US:name":"Desert Eagle",
"lng:en_US:in_stock": true,
"lng:en_US:url": "https://www.shop.com/en_US/products/desert-eagle",
"lng:en_US:image_url": "//images/en_US/shoe.png",
"lng:en_US:display_price":"200$",
"lng:en_GB:name":"Desert Eagle",
"lng:en_GB:in_stock": false,
"lng:en_GB:url": "https://www.shop.com/en_GB/products/desert-eagle",
"lng:en_GB:image_url": "//images/en_GB/shoe.png",
"lng:en_GB:display_price":"200£",
"lng:fr_FR:name":"Aigle du désert",
"lng:fr_FR:in_stock": true,
"lng:fr_FR:url": "https://www.shop.com/fr_FR/products/desert-eagle",
"lng:fr_FR:image_url": "//images/fr_FR/shoe.png",
"lng:fr_FR:display_price":"200€",
"lng:de_DE:name":"Wüstenadler",
"lng:de_DE:in_stock": true,
"lng:de_DE:url": "https://www.shop.com/de_DE/products/desert-eagle",
"lng:de_DE:image_url": "//images/de_DE/shoe.png",
"lng:de_DE:display_price":"200€"
}]
sku,group_id,name,in_stock,price,image_url,url,categories
111,22,t_shirt,true,9.50,http://dy.com/shop/111/ex.jpg,https://www.dy.com/shop/111.html?ch=22,mens|sport
"sku","group_id","categories","name","url","price","image_url","in_stock","keywords","display_price","gender","product","surface","best_for_1","best_for_2","color_1","color_2","cushioning","support","lng:en_US:name","lng:en_US:in_stock","lng:en_US:url","lng:en_US:image_url","lng:en_US:display_price","lng:en_GB:name","lng:en_GB:in_stock","lng:en_GB:url","lng:en_GB:image_url","lng:en_GB:display_price","lng:fr_FR:name","lng:fr_FR:in_stock","lng:fr_FR:url","lng:fr_FR:image_url","lng:fr_FR:display_price","lng:de_DE:name","lng:de_DE:in_stock","lng:de_DE:url","lng:de_DE:image_url","lng:de_DE:display_price"
"1125-blue-42","1125","shoes|mens|road","Desert Eagle","https://www.shop.com/en_US/products/desert-eagle",225,"//images/en_US/shoe.png",true,"active|training|black|grey|direct|neutral","225$","mens","shoes","road","active","training","black","grey","direct","neutral","Desert Eagle",true,"https://www.shop.com/en_US/products/desert-eagle","//images/en_US/shoe.png","200$","Desert Eagle",false,"https://www.shop.com/en_GB/products/desert-eagle","//images/en_GB/shoe.png","200£","Aigle du désert",true,"https://www.shop.com/fr_FR/products/desert-eagle","//images/fr_FR/shoe.png","200€","Wüstenadler",true,"https://www.shop.com/de_DE/products/desert-eagle","//images/de_DE/shoe.png","200€
Step 2: Sync the file with Experience OS
If your feed file contains more than 20,000 items, contact your Technical Account Manager to sync your feed. If your feed is smaller, go to Assets › Data Feeds and create a new product feed.
After naming the feed, you can select one of the following sync options:
Sync a URL or upload a file
Upload a feed file (up to 25 MB), or refer to a URL location where a feed file is stored. If you are using HTTP authentication, add the username and password to the feed URL as follows http://user:password@url.
You can click Add Another Source to combine up to 10 sources:
You can then use a parser function to control how the two feed files are merged, as seen in the following example:
function parse(feed1, feed2) {
var combinedFeeds = feed1.concat(feed2);
return combinedFeeds;
} Sync securely via Amazon S3
Use a password-protected S3 bucket managed by Dynamic Yield that contains your product feed file.
- Click Create a Bucket to create a new bucket. The credentials are only displayed once, so make sure you copy them to a secure location. You can generate new credentials using the Generate New Credentials option, but this is also limited to two times. Experience OS checks the bucket for a new file six times a day. If you just uploaded a file to the bucket, you can also click Sync Now to force the initial sync.
- To perform the initial upload to the bucket, use the AWS CLI tool as seen in the following example for a US account (EU endpoints are different and provided in the Experience OS console upon feed creation). Note that the filename must be productfeed.csv.
- Push the file to S3:
aws s3 cp local file .csv s3://com.dynamicyield.feeds/SECTION_ID/productfeed.csv - Verify that the file is in S3:
aws s3 ls s3://com.dynamicyield.feeds/SECTION_ID/
Make sure to add “/” at the end of the endpoint to avoid “access denied error”.
Sync securely via SFTP
Use an S3 bucket, secured using SFTP protocol, managed and provided by Dynamic Yield, which contains your product feed file.
- Enter your SSH key to register it with Experience OS (only one can be registered per OS site at any time).
- Use the Endpoint, Server ID, and username to upload your feed (see screenshot). The filename must be productfeed.csv. You can change the SSH key at any time using the Change Public SSH Key link.
- Experience OS checks the bucket for a new file six times a day. If you just uploaded a file to the bucket, you can also click Sync Now to force the initial sync.
Sync frequency
By default, the product feed syncs with the source file every 24 hours. However, you can change this setting if your product catalog updates more frequently. Make the adjustment in Advanced Settings. If you sync the file via S3 or SFTP, Experience OS checks the bucket for a new file six times a day. If you just uploaded a file to the bucket, you can also click Sync Now to force the initial sync.
Alternatively, you can use an API to update the feed on demand. This option enables you to add, delete, or modify entire rows or values in your data feed and sync it more quickly. However, this method requires advanced setup and technical support. Contact your Technical Account Manager for more details.
Step 3: Parse the source file (optional)
If the columns and values in your feed file don’t meet the requirements above, or if some other manipulation is required in order to duplicate, move around or rename values and fields, you can modify the feed using a parser function created in JavaScript. For example, XML files which are not "flat" can be "flattened" using a parser function.
Note: Parser functions are only supported for data feeds up to 200,000 items and 180 MB.
To add a parser function, go to Advanced Settings › Parser Function and enter your code.
Parser functions are required for XML feeds so the feed can be converted into an array inside a JSON.
function parser(data){
var products = data.items[0].item;
return products.map(function(item){
var newFeedRow = {};
for(var column in item){
newFeedRow[column] = item[column][0];
}
return newFeedRow;
});
}You can use a parser function to combine two product feeds into one feed.
function parse(feed1, feed2) {
var combinedFeeds = feed1.concat(feed2);
return combinedFeeds;
} GPF (Google Product Feed), commonly used for Shopping Ads, is very similar to the Experience OS product feed structure. If you already have a GPF, you can use the following parser function code as an example, and adjust it as needed.
function parse(data) {
const DY_GOOGLE_FIELD_MAPPING = {
'sku': 'g:id',
'name': 'g:title',
'url': 'g:link',
'price': 'g:price',
'in_stock': 'g:availability',
'image_url': 'g:image_link',
'categories': 'g:product_type',
'group_id': 'g:item_group_id'
};
/*
in case the category is constructed from several feeds with the same name and a number,
for example: "g:category_level1","g:category_level2","g:category_level3":
*/
const DY_GOOGLE_FIELD_MAPPING_CATEGORIES_FALLBACK = 'g:category_level*';
const MAX_FALLBACK_CATEGORIES = 10;
const TRY_FINDING_ITEMS_IN_XML_AUTOMATICALLY = true;
const ONLY_ADD_ITEMS_FROM_FIELD_MAPPING = false;
/* In case the automatic item allocation is not used:*/
const FEED_ITEMS_ARRAY_PATH = data.rss[0].channel[0].item;
const parseItem = oldItem => {
let newItem = {};
for (let dyFieldName in DY_GOOGLE_FIELD_MAPPING) {
if (!DY_GOOGLE_FIELD_MAPPING.hasOwnProperty(dyFieldName)) {
continue;
}
switch (true) {
case dyFieldName === 'sku':
case dyFieldName === 'name':
case dyFieldName === 'url':
case dyFieldName === 'image_url':
case (dyFieldName === 'group_id' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.group_id] !== 'undefined'):
newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]);
break;
/* if no group id found in the original feed, use the sku instead:*/
case (dyFieldName === 'group_id' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.group_id] === 'undefined'):
newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING.sku);
break;
case dyFieldName === 'in_stock':
newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]) === 'in stock';
break;
case dyFieldName === 'price':
newItem[dyFieldName] = parseFloat(stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]).replace(',', '.').replace(/[^0-9\.-]+/g, ''));
break;
case (dyFieldName === 'categories' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.categories] !== 'undefined'):
newItem[dyFieldName] = stringifyValue(oldItem, DY_GOOGLE_FIELD_MAPPING[dyFieldName]).replace(/\ >\ /g, '|');
break;
/* if the category feed doesn't exist, loop through the fallback value*/
case (dyFieldName === 'categories' && typeof oldItem[DY_GOOGLE_FIELD_MAPPING.categories] === 'undefined'):
let categoryStr = '';
for (let i = 0; i < MAX_FALLBACK_CATEGORIES; i++) { let categoryFieldName = DY_GOOGLE_FIELD_MAPPING_CATEGORIES_FALLBACK.replace('*', i); if (oldItem[categoryFieldName] && oldItem[categoryFieldName][0]) { if (categoryStr.length) { categoryStr += '|'; } categoryStr += oldItem[categoryFieldName][0]; // delete oldItem[categoryFieldName]; } if (categoryStr.length && typeof oldItem[categoryFieldName] === 'undefined') { break; } } newItem[dyFieldName] = categoryStr; break; default: } } if (!ONLY_ADD_ITEMS_FROM_FIELD_MAPPING) { for (let customerField in oldItem) { if (!oldItem.hasOwnProperty(customerField)) { continue; } let fieldValue = stringifyValue(oldItem, customerField); if (typeof fieldValue === 'object') { // this currently support nested objects for non mandatory/mapped fields. parseNestedObject(newItem, oldItem, customerField); } else { newItem[customerField] = fieldValue; } } } return newItem; }; /* this funtion scans the xml for array larger than 1, which will most likley only be the array of items. */ const getItemsNode = feed => {
let elements = [feed];
for (let maxTries = 100; maxTries > 0; maxTries--) {
if (!elements.length) {
break;
}
let element = elements.pop();
if (typeof element !== 'object') {
continue;
}
if (element.length && element.length > 1) {
return element;
}
if (element.length) {
elements.push(element[0]);
} else {
for (let child in element) {
if (!element.hasOwnProperty(child)) {
continue;
}
elements.push(element[child]);
}
}
}
return true;
};
/* tries to return the value as a string, else return the value as is*/
const stringifyValue = (oldItem, field) => {
if (!oldItem[field]) {
return '';
}
let fieldValue = oldItem[field][0];
if (typeof fieldValue !== 'object') {
return (fieldValue + '').replace('<![CDATA[', '').replace(']]>', '');
}
return fieldValue;
};
/*
some google fields contains nested objects. This only covers one child hirarchy and
naming the nested field in this patern: parent_child_i for example: 'g:shipping_g:country_1'
*/
const parseNestedObject = (newItem, oldItem, parentFieldName) => {
for (let i = 0; i < oldItem[parentFieldName].length; i++) {
let nestedFieldObject = oldItem[parentFieldName][i];
for (let nestedField in nestedFieldObject) {
if (!nestedFieldObject.hasOwnProperty(nestedField)) {
continue;
}
newItem[parentFieldName + '_' + nestedField + '_' + i] = nestedFieldObject[nestedField][0];
}
}
};
let oldItems;
if (TRY_FINDING_ITEMS_IN_XML_AUTOMATICALLY) {
oldItems = getItemsNode(data);
} else {
oldItems = FEED_ITEMS_ARRAY_PATH;
}
return oldItems.map(parseItem);
}Step 4: Save and preview
If your feed source is an uploaded file or URL, you can view a preview of up to 100 entries of your feed including any errors or warnings by clicking Preview. Click Save and Activate to proceed or Cancel to go back.
The synchronization duration depends on the number of items in your feed. For feeds with 250,000 items in the feed it usually takes a few minutes, and 30 minutes for every additional 500,000 items in your feed. If you require a faster sync, consider updating your feed by API to update incremental changes instead of the standard full-feed update.
Step 5: Validate sync status
You can verify that your feed is synchronizing successfully at any time:
- Go to Assets › Data Feeds.
- In the Last Sync Attempt column, you should see a Date Feed Synced icon
and a recent date. If the icon indicates an error or warning, download the log for more details. - In the Actions column, click the View icon
. Validate that all required columns and any custom columns you have added exist and have the correct data.
Note: For feeds with more than 2 million products, products with no associated events are filtered out of the data feeds preview. Active products are products that have one or more activities (product view/add-to-cart/purchase) in the last 6 months. So when you view the product feed in the preview, the number of products at the bottom of the page is the number updated with this filter.