Event-driven incremental ingestion pipeline on Databricks with Delta Lake. Triggers on new file arrivals, loads data to staging, and performs SCD1 merges from stage into target tables—enabling ...

🚀 Project Overview This project demonstrates a complete data migration pipeline from a simulated on-premises source to Azure Cloud, utilizing the Medallion Architecture (Bronze, Silver, Gold). It ...