Skip to main content

Append Overview

The append process is highly modular. Each module will perform a different append process / validation on the data in the file. This page outlines the different modules and the name of the config key that needs to be present. The first part of the process is to create offers from information contained within the Design File. Once offers have been created, the process then moves onto appending pricing and performance data.

This data is sourced from Promo Files determined by country, brand and campaign period information specified in the Design File.

Section 1: General Configurations

  • GENERAL_CONFIGS: Defines general settings for the append process.
  • LOGGING_CONFIGS: Specifies detailed logging instructions for specific modules. All output does get logged, these options allow the user to specifiy which logs are visible in the application when the process runs.

Section 2: Offer Processing

  • NewOfferIDProcessing: These configurations specifies how offers are constructed and grouped so that they are in alignment with internal processes.

Section 3: Offer Pricing Calculation

  • SecondaryKeyAverage, SecondaryKeyCount: Calculates secondary key averages and counts based on specific fields.
  • SecondaryKeyFields: Lists secondary key fields, including element keys and price sets. In most cases the primary keys are the brand family, however because a brand family could appear in multiple offers we need a secondary key to identify the correct instance of the item. This is the Secondary Key field. For the most part we create a field called an OFFER HASH, this field joins and hashes the contents of an offer.

Section 4: Appending Meta Data

  • Prefixes: Defines prefixes for certain fields, such as "ANY=" and "DOUBLE=".

Section 5: Data Processing Configurations

  • WorkingPageFix: Fixes issues related to working pages, including mapping from old page splits to new ones.
  • AssignBlockIndex: Assigns block indices to certain fields based on a cluster field name and field-to-index map.
  • PromoKeyProcessing: Processes promo keys by joining with a specific join field and keying on an element key.
  • DesignKeyProcessing: Similar to promo key processing, but for design keys.
  • PageReferenceProcessing: Processes page references by counting pages by country code.
  • CalculateMax: Configures calculations for maximum values of various fields.

Section 4: Post-Processing Configurations

  • PostProcessing: Applies post-processing rules to the data, such as human-readable reference extraction from source file names.

  • TonesAndShades: Defines configurations for processing tones and shades data.

  • Prefixes, CalculateMax: Same as in Section 1, but with additional configurations.

Section 6: Data Validation Configurations

  • ValidationPolicy: Defines validation policies for various fields, including discount and offer type value checks. These are run at the very end of the process and can be applied to any field.

This configuration file is used in a data pipeline that processes and transforms data for various business use cases. The configurations cover topics such as data validation, post-processing, and transformation of specific fields like promo keys, design keys, page references, tones and shades, and secondary key averages and counts.