Teaser .. Metadata driven framework in ADF

In lack of any actual content as of right now, I thought I might as well add a little teaser for an upcoming series of blog posts regarding some recent work building a metadata driven framework in Azure Data Factory.

Keywords:

  • Data Factory orchestration pattern
  • Metadata repository feeding a set of pipelines and mapping data flows supporting
    • Multiple projects (Ingest, DW, etc)
    • Various source systems
    • Full and incremental load patterns
    • Integration stages for sequential orchestration
    • A selection of slowly changing dimensions patterns
    • Variable number of reference tables and columns
    • Where and Transform operations on columns
  • Ingest from various source systems to parquet files in Azure Data Lake using ADF pipelines with auto-creation of views in Synapse
  • Transformation from raw staging area in Data Lake (exposed through views in Synapse) to 3NF modelled Data Warehouse in Azure Synapse (SQL DW)
  • Logging
  • Monitoring