In lack of any actual content as of right now, I thought I might as well add a little teaser for an upcoming series of blog posts regarding some recent work building a metadata driven framework in Azure Data Factory.
Keywords:
- Data Factory orchestration pattern
- Metadata repository feeding a set of pipelines and mapping data flows supporting
- Multiple projects (Ingest, DW, etc)
- Various source systems
- Full and incremental load patterns
- Integration stages for sequential orchestration
- A selection of slowly changing dimensions patterns
- Variable number of reference tables and columns
- Where and Transform operations on columns
- Ingest from various source systems to parquet files in Azure Data Lake using ADF pipelines with auto-creation of views in Synapse
- Transformation from raw staging area in Data Lake (exposed through views in Synapse) to 3NF modelled Data Warehouse in Azure Synapse (SQL DW)
- Logging
- Monitoring