Arnab Saha

The project aimed to migrate the Assurant Specialty Property business from the existing system to a new integrated and modernized single-source platform. The business target was to increase revenue by 32%. The application offered a single platform to issue and manage insurance policies, help users pay premiums, and submit claims and beneficiaries. It covered 85% of Assurant Property's functional areas and automated its business process to eliminate human interventions.

I was a part of the team that got daily files of various sizes from different clients and created four logical divisions to process the data (pre-stage, stage, native, and SSP). We also verified the file quality and loaded the whole file in Prestage in a single column with different tables for each client. Then we bifurcated data, loaded the columns with Truncate, and pushed data. Finally, we performed various business transformations and loaded them into a single SSP layer. We used a logging framework to log the status of each pipeline to track it correctly. During the project delivery, we followed the Agile model, TFS/Azure DevOps as source control, and CI/CD process for deployment.

The client had implemented a monitoring system and stored data mainly in Hive tables. The client wanted to move their data processing layer to a cloud environment, so I helped them move their entire data processing layer to the Azure cloud environment.
To migrate the whole processing layer, we mainly used the following cloud resources:
1. Azure Data Factory
2. Azure Databricks
3. Azure Data Lake Storage
4. Azure Event Hub
5. Azure Application Insights and Log Analytics

They have some streaming data sources (in Avro format), RDBMS tables, and File storage. We used Event Hub and Databricks to handle streaming and Data Factory as orchestration tools. We used a Databricks Notebook for all transformations activities, and using Notebook activity in ADF, invoked Databricks Notebook. We have been used Pipeline triggers (dependent triggers) and the scheduled trigger.

For the storage, we mainly used ADLS and created three logical divisions in the ADLS—stage, discovery, and insight layer. Through the insight layer, we connected with reporting tools for different insights and visualizations

The client was one of the reputed names in the retail domain. They have different channels online and offline like merchandise, distribution, and others to sell their product. They want to have a unified data layer and reports from where they can get a quick idea of the overall business summary and insights.
We designed and created a data warehouse (OLAP system) to store data from different channels in a unified fashion and used SQL (Azure SQL) as a storage model.
We have created different DB from the stage, native, and dimension modeling. Also, we created an audit framework to handle the error. As data size was massive, we created a separate tabular model and hosted it in the Azure Analysis service.
We connected the Power Bi report (Live Connection) with AAS, created different visuals for reporting purposes, and hosted in Power Bi service.

We use ADF as an orchestration pipeline, which will get the data from on-premise sources, then process through SP-Activity and refresh tabular model in AAS and send successful/failure alert through Azure Logic App.