Teaser .. Metadata driven framework in ADF

In lack of any actual content as of right now, I thought I might as well add a little teaser for an upcoming series of blog posts regarding some recent work building a metadata driven framework in Azure Data Factory.

Keywords:

  • Data Factory orchestration pattern
  • Metadata repository feeding a set of pipelines and mapping data flows supporting
    • Multiple projects (Ingest, DW, etc)
    • Various source systems
    • Full and incremental load patterns
    • Integration stages for sequential orchestration
    • A selection of slowly changing dimensions patterns
    • Variable number of reference tables and columns
    • Where and Transform operations on columns
  • Ingest from various source systems to parquet files in Azure Data Lake using ADF pipelines with auto-creation of views in Synapse
  • Transformation from raw staging area in Data Lake (exposed through views in Synapse) to 3NF modelled Data Warehouse in Azure Synapse (SQL DW)
  • Logging
  • Monitoring

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>