Exporting Raw Data Using the Daily Activity Stream

Follow

You can export the data that Dynamic Yield tracks, and ingest it into your BI platform. The export is done on a daily basis, in a Parquet file that is exported to a secured S3 bucket. The data in the S3 bucket is stored for 30 days due to privacy regulations. 

This data allows your BI team to:

  • Build a customized report that is tailored to your unique business needs.
  • Connect this data to additional data sources and build reports. For example, join CRM data and acquisition performance and build more holistic reports. 

Data Format

The file is in Apache Parquet format that is optimized for large data sets. The file name will contain the date, and the file content will include all of your traffic activities (one entry per activity). 

There are 3 types of activities, with a different eventType attribute value, and different attributes according to the type :

eventType Description Additional Attributes (Examples)
UIA Pageview URL, page context
DPX Event hit Event properties, event value
VARIATION_ENGAGEMENT Variation impression or click Campaign name, variation name

These are the full list of attributes according to the eventType:

eventType: UIA (Pageview)

Represents a pageview that the user has made.

Attribute Description Example
eventType
String		 The type of activity. For pageview - value is UIA
(as opposed to VARIATION_ENGAGEMENT or DPX)		 "UIA"
contextType
String		 The page type, according to the page context. "HOMEPAGE"
contextData
String array

The data about the page type.

  • If page type is PRODUCT - the SKU of the page.
  • If page type is CATEGORY - the category name.
  • If page type is CART - the SKUs of all products in the cart.
 ["Women","Shoes"]
dyId
Long		 The internal identifier Dynamic Yield assigns to each visitor in the site or app, unique per device. 1.23456789098765E+018
timestamp
Long		 The time of the activity has occurred, in milliseconds from the UNIX epoch. 1621798861400
sectionId
Integer		 The Dynamic Yield site ID as assigned in your Dynamic Yield account. 123456
eventType
String		 The type of event fired. It is always one of the following:
DPX: An event using one of Dynamic Yield’s event APIs
UIA: A pageview
VARIATION_ENGAGEMENT: The engagement with a Dynamic Yield Variation
UNIT_IMPRESSION: Check
UNIT_CLICK: Check		 "DPX"
sessionId
Integer		 The Internal identifier Dynamic Yield assigns to a visitor's session 1234567890
url
String		 The URL from which the event was fired. "https://www.example.com/?url_params=123"
urlClean
String		 The URL from which the event was fired, after removing any URL parameters. "https://www.example.com/"
audiences
Integer array		 The list of audience identifiers the user is a member of at the time of firing the event. [1234567, 9876543]
browser
String		 The browser type from which the event was fired. "Safari"
device
String		 The type of device which triggered the event "Tablet"
operatingSystem
String		 The operation system of the device that fired the event. "Mac OS X"
screenResolution
String		 The screen resolution of the device that fired the event. "Low (1024px and below)"
reqTimestamp
Long		 Internal request timestamp to the analytics pipeline, in milliseconds from the UNIX epoch. 1621798861400
procTimestamp
Long		 Internal processing timestamp of the analytics pipeline, in milliseconds from the UNIX epoch. 1621798861400
resTimestamp
Long		 Internal resolution timestamp of the analytics pipeline, in milliseconds from the UNIX epoch. 1621798861400
eventType: DPX (Event)

Represents a Dynamic Yield event that the user has fired.

Attribute Description Example
eventType
String		 The type of activity. For event hit - value is DPX
(as opposed to VARIATION_ENGAGEMENT or UIA)		 "DPX"
eventId
Integer		 A unique identifier for each event explicitly fired from the site. 12345
eventName
String		 The event name as written in the event API. "Purchase"
eventProperties
String (JSON)		 The event properties as written in the event API. These differ depending on the eventType value. {
"transaction_id": "ABC123456",
"value": 100.0,
"currency": "USD",
"dyType": "purchase-v1",
"Brands": "Nike",
"Categories": "Sneakers",
"Number_of_items": 1.0,
"cart": [{
"productId": "AIR-123",
"quantity": 1.0,
"itemPrice": 100.0
}]
}
eventValue
Long		 The monetary value in cents of an event as written in the event API. 1000
uniqueTransactionId
String		 A unique transaction id for purchase events, as written in the purchase event API. "ABC123456"
dyId
Long		 The internal identifier Dynamic Yield assigns to each visitor in the site or app, unique per device. 1.23456789098765E+018
timestamp
Long		 The time of the activity has occurred, in milliseconds from the UNIX epoch. 1621798861400
sectionId
Integer		 The Dynamic Yield site ID as assigned in your Dynamic Yield account. 123456
sessionId
Integer		 The Internal identifier Dynamic Yield assigns to a visitor's session 1234567890
url
String		 The URL from which the event was fired. "https://www.example.com/?url_params=123"
urlClean
String		 The URL from which the event was fired, after removing any URL parameters. "https://www.example.com/"
audiences
Integer array		 The list of audience identifiers the user is a member of at the time of firing the event. [1234567, 9876543]
browser
String		 The browser type from which the event was fired. "Safari"
device
String		 The type of device which triggered the event "Tablet"
operatingSystem
String		 The operation system of the device that fired the event. "Mac OS X"
screenResolution
String		 The screen resolution of the device that fired the event. "Low (1024px and below)"
reqTimestamp
Long		 Internal request timestamp to the analytics pipeline, in milliseconds from the UNIX epoch. 1621798861400
procTimestamp
Long		 Internal processing timestamp of the analytics pipeline, in milliseconds from the UNIX epoch. 1621798861400
resTimestamp
Long		 Internal resolution timestamp of the analytics pipeline, in milliseconds from the UNIX epoch. 1621798861400
eventType: VARIATION_ENGAGEMENT (Click or Impression)

Represents an impression of a variation or a click on a variation.

Attribute Description Example
eventType
String		 The type of activity. For variation click or impression, the value is VARIATION_ENGAGEMENT
(as opposed to UIA or DPX)		 "VARIATION_ENGAGEMENT"
engagementType
String

The type of engagement with the variation. Possible values:

  • "IMPRESSION"
  • "CLICK"
 "IMPRESSION"
campaignId
Integer		 The ID of the campaign that this variation is part of. 123456
campaignName
String		 The name of the campaign that this variation is part of. "Homepage Banner"
experienceId
Integer		 The ID of the experience that this variation is part of. 123456
experienceName
String		 The name of the experience that this variation is part of. "Summer Promo"
experimentId
Integer		 The unique identifier of the test version. 123456
versionId
Integer		 The unique identifier of the test version. An A/B test might have multiple versions. 245467
variationIds
Integer array		 The ID of the variation that the user was served with (if the type is IMPRESSION) or clicked on (if the type is CLICK).
In most cases, the list contains a single ID, but if the campaign type is "Dynamic Content Item List", it will contain a list of variations, separated with a comma.		 [1234567,
9876543]
variationNames
String array		 The name of the variation that the user was served (if the type is IMPRESSION) or clicked on (if the type is CLICK).
Usually, the list contains a single ID, but if the campaign type is "Dynamic Content Item List", it contains a list of variations, separated by commas.		 ["Blue Button", "Red Button"]
sku
String array

If the event is viewing or clicking a recommendation widget, it's the list of SKUs that were recommended, or the SKU that was clicked.

 ["1234",
"9876"]
strategyId
Integer array

If the event is viewing or clicking a recommendation widget, it's the ID of the Strategy that was served.

A single variation can include multiple widgets with multiple strategies.

 [126651,426356]

strategyName
String array

If the event is viewing or clicking a recommendation widget, it's the name of the Strategy that was served.

A single variation can include multiple widgets with multiple strategies.

 ["Most Popular","Affinity"]
touchpointId
Integer
 In touchpoints only: The ID of the touchpoint. 245467

touchpointName
Integer

 In touchpoints only: The name of the touchpoint. ["Hero Banner"]
parentVariationId
Integer
 In touchpoints only: The ID of the variation that serves this touchpoint in the multi-touch campaign. [9876543]

parentVariationName
Integer

 In touchpoints only: The name of the variation that serves this touchpoint in the multi-touch campaign. "Blue Design"
dyId
Long		 The internal identifier Dynamic Yield assigns to each visitor in the site or app, unique per device. 1.23456789098765E+018
timestamp
Long		 The time of the activity has occurred, in milliseconds from the UNIX epoch. 1621798861400
sectionId
Integer		 The Dynamic Yield site ID as assigned in your Dynamic Yield account. 123456
sessionId
Integer		 The Internal identifier Dynamic Yield assigns to a visitor's session 1234567890
url
String		 The URL from which the event was fired. "https://www.example.com/?url_params=123"
urlClean
String		 The URL from which the event was fired, after removing any URL parameters. "https://www.example.com/"
audiences
Integer array		 The list of audience identifiers the user is a member of at the time of firing the event. [1234567, 9876543]
reqTimestamp
Long		 Internal request timestamp to the analytics pipeline, in milliseconds from the UNIX epoch. 1621798861400
procTimestamp
Long		 Internal processing timestamp of the analytics pipeline, in milliseconds from the UNIX epoch. 1621798861400
resTimestamp
Long		 Internal resolution timestamp of the analytics pipeline, in milliseconds from the UNIX epoch. 1621798861400

Turning On the Daily Activity Stream

  1. Go to Settings › General Settings › Daily Activity Stream.
    daily_activity_stream.png
  2. Click Turn on daily export.
  3. Copy the S3 bucket and credentials to a secure location as it is only displayed once.image2.png

    Lost your credentials?
    Click the additional options icon (⋮) and click Generate New Credentials. Notice that you can only generate new credentials only once.

     

  4. That's it! The export is prepared on a daily basis, exported to the S3 buckets during the night.

After turned on, you can see the status of the export.
You can always disable the export by clicking the options icon (⋮). and If you disable the export, and then enable it again, your S3 bucket will stay the same, but the credentials will change. 

Importing Data Into Your Analytics Platform

In order to analyze data based on Daily Activity Steam, you need to ingest the Parquet file from the S3 folder into the data warehouse used by your Analytics platform. Refer to the documentation of your data warehouse for more information. Here are some links to the relevant articles of some common ones: