One of the largest Payers in North America (Fortune 500)
Business Challenge
Enhancing control over its Data Lake by migrating it to Azure, was the key intention behind this project. The client wanted to create a modernized, easily accessible, and scalable data landscape. The migration aimed to bring the Data Lake closer to source systems and reporting platforms, which would isolate data workloads for improved control and efficiency.
- Streamlined 21 domains and enhanced workflow orchestration, heightening efficiency in a large Payer’s operations.
- Initiated the migration of 21 domains to Azure, integrating L0 components, including the development of a robust Autoloader Framework for diverse file types.
- Key activities encompassed:
- External vendor file ingestion, QDRM, Claims Workflow, MCG, Edge Server, and ODS ingestion.
- Creating a data quality dashboard.
- Artifacts detailing workflow specifics were generated, and a secured view with data automation capabilities was established.
- Automation of the data pipeline development.
- Implemented ingestion framework, providing workflow orchestration, time driven automation, workflow parallelize, data flow management, scheduling, monitoring, visibility and alerting.
- 8Hrs Reduction in manual effort to create and load 1500 tables
- 12K Tables migrated from an on-premises system to Azure Databricks using BDR Framework
