Teaser .. Metadata driven framework in ADF

In lack of any actual content as of right now, I thought I might as well add a little teaser for an upcoming series of blog posts regarding some recent work building a metadata driven framework in Azure Data Factory.

Keywords:

  • Data Factory orchestration pattern
  • Metadata repository feeding a set of pipelines and mapping data flows supporting
    • Multiple projects (Ingest, DW, etc)
    • Various source systems
    • Full and incremental load patterns
    • Integration stages for sequential orchestration
    • A selection of slowly changing dimensions patterns
    • Variable number of reference tables and columns
    • Where and Transform operations on columns
  • Ingest from various source systems to parquet files in Azure Data Lake using ADF pipelines with auto-creation of views in Synapse
  • Transformation from raw staging area in Data Lake (exposed through views in Synapse) to 3NF modelled Data Warehouse in Azure Synapse (SQL DW)
  • Logging
  • Monitoring

Hello world!

Well .. here goes! First blog post ever.

The reason for stepping into the great unknown of blogging is partly for my own documentation of thoughts, ideas, projects and solutions within BI & Analytics and partly for sharing the same things with anyone who might find inspiration or even answers to their problems through my ramblings.

To whomever might stumble upon this blog – I hope you find the content useful and if so, please drop a comment or reach out.