Kiran is currently unavailable
Kiran Kumar Kolla
Verified Expert in Engineering
Database Engineer and Developer
Sydney, New South Wales, Australia
Toptal member since July 28, 2021
Bio
Kiran is a database engineer with over ten years of expertise implementing data lakes, data integrations, and data warehousing solutions. In addition to data engineering work with Cognizant and wireless with NBN Co, Kiran has enterprise experience in financial services from MetLife, ING Group, to Bank of America. He is highly motivated when challenges are thrown at him, as they bring out his best! Kiran loves to program and use logical thinking to solve complex problems efficiently.
Portfolio
Hollard Insurance
Azure, Azure Databricks, Azure SQL, Python 3, PySpark, Azure Data Factory (ADF)...
B.S.D. Capital, Inc. dba Lendistry
Data Engineering, Python, Scala, Java, SQL, GitHub, Bitbucket, Apache Airflow...
Arq Group
Azure Databricks, Azure Cosmos DB, Azure Event Hubs, Azure Synapse Analytics...
Experience
- Python 3 - 4 years
- Databricks - 4 years
- Azure Synapse Analytics - 3 years
- Azure DevOps - 2 years
- Azure Functions - 2 years
- Azure SQL - 2 years
- Azure Data Factory (ADF) - 2 years
- Microsoft Power Apps - 1 year
Work Experience
Senior Data Engineer
2022 - 2026
Hollard Insurance
- Implemented data ingestion framework for the Hollard data platform as part of the Finance Data Transformation project.
- Built data pipelines to ingest the policy, claims, and finance data from partners into the data lake using Azure Data Factory, Databricks, and Azure Synapse.
- Created DevOps pipelines to automatically promote the code for Data Factory, Databricks, and SQL database to higher environments.
- Developed Power BI dashboards to track data ingestion issues from multiple partners and designed domain-specific error flag reports (policy, claims, collections) to identify root causes of variances between expected and actual values.
Technologies: Azure, Azure Databricks, Azure SQL, Python 3, PySpark, Azure Data Factory (ADF), Data Pipelines, Delta Lake, Azure DevOps, Azure Logic Apps, Microsoft Power BI, Power BI Desktop, Data Engineering, Data Analysis, Business Intelligence (BI), Microsoft Fabric, Azure Synapse
Senior Data Engineer (via Toptal)
2022 - 2024
B.S.D. Capital, Inc. dba Lendistry
- Automated the process of combining the events related to loan applications into a single table using Python, dbt, and Airflow, which simplified the reporting process and saved much manual effort.
- Moved all the complex SQL logic into dbt workflows and created Airflow DAGs on top. Created reusable macros in dbt (test cases, functions) to dynamically generate the models.
- Migrated several existing KNIME workflows into Python scripts to automate the process of Excel report generation as per business requirements and trigger automated emails.
- Automated the approval process of adding items to SharePoint lists (a data dictionary) using Power Automate, which sends notifications to Teams groups for review/approval.
Technologies: Data Engineering, Python, Scala, Java, SQL, GitHub, Bitbucket, Apache Airflow, ETL, Amazon Web Services (AWS), KNIME, RDBMS, Data Warehousing, Data Processing, Amazon Redshift, Data Build Tool (dbt), Lambda Functions, Amazon Managed Workflows for Apache Airflow (MWAA), Power Automate Cloud Flow, Python 3, Databricks, Amazon API Gateway
Senior Data Consultant
2022 - 2023
Arq Group
- Provided solution design for various enterprise data capabilities using Azure Cloud Services and batch and stream processing architecture.
- Created Databricks streaming job to ingest the data from Cosmos DB using change feed and push the data to Salesforce APIs in real time.
- Coordinated with business analysts, data modelers, developers, and enterprise architects and proposed the end-to-end architecture that meets the requirements and enterprise standards.
Technologies: Azure Databricks, Azure Cosmos DB, Azure Event Hubs, Azure Synapse Analytics, Azure Cloud Services, Streaming Data, Microsoft SQL Server, ETL Tools, Salesforce REST API, Data Engineering
DevOps Engineer
2021 - 2022
AB-InBev - Israel
- Created the automated release pipelines for Azure SQL SB of each environment.
- Maintained and supported the Azure infrastructure for SQL databases, Data factory, etc.
- Automated the process of Azure Data Factory deployment for different teams across environments through Azure DevOps.
Technologies: Azure, Azure Data Factory (ADF), Data Pipelines, Azure Data Lake, Azure DevOps, Azure SQL, Azure Cloud Services, Microsoft SQL Server
Senior Data Engineer
2020 - 2022
Data-Driven AI
- Developed Azure durable entity functions to ingest data from Open Data Hub APIs in case of changes in the response header. The action saved 8GB of storage space every day and reduced costs significantly.
- Built an app in Microsoft Power Apps to fetch resource usage costs and cost optimization recommendations from an Azure subscription. The app helped users monitor resources and reduce costs.
- Created a self-service API and Microsoft Power app for business users to securely download historical GTFS data on-demand from a data lake without manual intervention. This action saved 40 hours of manual effort every month.
- Built data pipelines using Azure Databricks (PySpark) to process the ingested data into the Delta Lake and the Synapse data warehouse.
- Created streaming data pipelines using Databricks to ingest data from Event Hubs and Cosmos DB.
- Created a Microsoft PowerApps application to enable business users to download historical vehicle incident data by category and timeframe.
Technologies: Azure Data Factory (ADF), Azure Synapse, Databricks, Microsoft Power Apps, Microsoft Power Automate, Azure Functions, Azure Logic Apps, Azure Automation, Azure DevOps, Data Lakes, Data Engineering, Python, C#, Microsoft Power BI, PySpark, Azure IaaS, Data Pipelines, Transact-SQL (T-SQL), Pandas, Data Cleansing, Power Automate Cloud Flow, Python 3, Azure Data Lake Analytics, Azure Cosmos DB, Azure Cloud Services, Microsoft SQL Server, ETL Tools, Power BI Desktop, Data Analysis
Data Engineer
2019 - 2020
NBN Co
- Developed automated ETL data pipelines using Spark on AWS EMR and Lambda to process the data, such as orders and services, from Comptel 7 and Maximo into an AWS S3 data lake.
- Created event-based Lambda functions using Python to trigger the Glue crawlers and metadata tagging.
- Developed data pipelines to transform order data from the UDS KPI model into business-friendly views that can easily be used for data analysis and reporting.
Technologies: Redshift, AWS Lambda, Alteryx, Tableau Desktop Pro, Amazon S3 (AWS S3), Apache Airflow, ETL, Amazon Elastic MapReduce (EMR), Spark, Data Lakes, Python, Tableau, Data Modeling, SQL, SQL Server Integration Services (SSIS), PySpark, Transact-SQL (T-SQL), Pandas, Data Visualization, Amazon Web Services (AWS), Data Cleansing, Python 3, ETL Tools, Data Engineering, Business Intelligence (BI)
Data Analyst
2018 - 2019
MetLife
- Created Alteryx workflows to extract data from multiple source systems and prepare the data for regulatory submission.
- Prepared the data for regulatory reporting (including APRA, ASIC, and LCCC) and generated reconciliation dashboards.
- Designed and developed Tableau functional dashboards based on business requirements.
Technologies: Alteryx, Tableau, SQL, Dashboards, Dashboard Design, Regulatory Reporting, Transact-SQL (T-SQL), Data Visualization, Data Cleansing, Microsoft SQL Server, ETL Tools, Data Analysis, Business Intelligence (BI)
Data Engineer
2017 - 2018
ING Group
- Implemented Common Reporting Standard (CRS) workflows to generate customer and account extracts.
- Developed an ETL framework using Informatica workflow, Windows PowerShell, and SQL to automate the batch scheduling process for the data warehouse.
- Created the data model for the personal loans and credit card products to load the transactional data.
Technologies: Informatica ETL, Data Warehouse Design, Python, Windows PowerShell, Data Warehousing, SQL, ETL, Dimensional Modeling, Transact-SQL (T-SQL), Microsoft SQL Server, ETL Tools, Business Intelligence (BI)
Senior Analyst
2013 - 2017
Bank of America
- Implemented an Informatica ETL COE operations reporting solution to help managers generate reports on platform usage, the number of sessions executed, and the volume of data processed.
- Automated the process of monthly metrics reports for application support and platform maintenance teams, using Excel VBA and Microsoft SQL Server, which saved a lot of manual effort.
- Completed the process of upgrading all the Informatica ETL COE platforms to v10.2 by automating the process with Jenkins and shell scripting.
Technologies: Informatica ETL, Shell Scripting, Tableau, ETL, Data Visualization, ETL Tools, Data Analysis
ETL Developer
2008 - 2013
Cognizant
- Developed automatic data reconciliation workflows to compare the data with control files received from the source system.
- Created Autosys workflows to set up job dependencies and schedules.
- Created stored procedures using Oracle PL/SQL to perform the data validation.
Technologies: Informatica ETL, Oracle, SQL Server Integration Services (SSIS), Autosys, Data Cleansing, Microsoft SQL Server, ETL Tools
Project History
Operational Data Lake for Transport
An operational data lake for transport that ingests and processes real-time trip updates and vehicle position data from IoT devices. I developed the data lake, which forms the data foundation for many business use cases as part of multimodal performance reporting.
Web Scraping
https://www.teamrankings.comA Python executable to extract the data from the TeamRanking website about on-the-day schedules and, for those teams, get the latest rankings for different categories. I created this using Pandas, openpyxl, and Beautiful Soup libraries to extract data from the NCAA website and generate an Excel file with all the teams and their strengths in different areas, which are color-coded.
Monitoring Application for Azure Cloud
A Microsoft Power App to monitor resources and their consumption costs across multiple subscriptions. I developed the app and created functions to fetch the resource and cost information from Azure Subscriptions used by the Power App, which helps the cloud admin monitor costs and the resources created.
Education
2002 - 2006
Bachelor's Degree in Computer Science
Jawaharlal Technological Universilty - Hyderabad, India
Certifications
JULY 2021 - PRESENT
Azure AI Fundamentals
Microsoft
DECEMBER 2020 - DECEMBER 2022
Azure Data Engineer Associate
Microsoft
SEPTEMBER 2020 - PRESENT
Azure Fundamentals
Microsoft
AUGUST 2018 - JANUARY 2021
Tableau Desktop Certified Associate
Tableau
Skills
Libraries/APIs
Pandas, PySpark, Azure Cognitive Services, Beautiful Soup, Salesforce REST API
Tools
Azure Logic Apps, Informatica ETL, Tableau Desktop Pro, Tableau, Microsoft Power Apps, Apache Airflow, Autosys, Microsoft Power BI, Power BI Desktop, Azure Automation, Azure Machine Learning, Amazon Elastic MapReduce (EMR), GitHub, Bitbucket
Languages
Python 3, SQL, Python, Transact-SQL (T-SQL), C#, Scala, Java
Frameworks
Windows PowerShell, Spark, Azure Bot Framework
Paradigms
ETL, Dimensional Modeling, Azure DevOps, DevOps, Business Intelligence (BI)
Platforms
Databricks, Azure, Oracle, Azure Synapse, Microsoft Power Automate, Azure Functions, Alteryx, AWS Lambda, Amazon Web Services (AWS), Power Automate Cloud Flow, Azure Synapse Analytics, Azure IaaS, Azure PaaS, KNIME, Azure Event Hubs, Microsoft Fabric
Storage
Azure SQL, Amazon S3 (AWS S3), Data Pipelines, Azure Cloud Services, Microsoft SQL Server, Data Lakes, SQL Server Integration Services (SSIS), Azure Cosmos DB, Redshift, RDBMS
Other
Azure Data Factory (ADF), Data Warehousing, Azure Data Lake Analytics, Software Development, Data Warehouse Design, Shell Scripting, Data Engineering, Data Architecture, Data Cleansing, Amazon Managed Workflows for Apache Airflow (MWAA), ETL Tools, Web Scraping, Dashboards, Dashboard Design, Data Modeling, Microsoft Data Transformation Services (now SSIS), Data Analysis, Data Visualization, Data Build Tool (dbt), Lambda Functions, Streaming Data, Cloud Security, Cloud Services, Data Analytics, Regulatory Reporting, Internet of Things (IoT), Excel 365, Data Processing, Amazon Redshift, Azure Databricks, Azure Data Lake, Delta Lake, Amazon API Gateway, Machine Learning, LangChain, Artificial Intelligence (AI)
Collaboration That Works
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
1
Share your needs
Discuss your requirements and refine your scope in a call with a Toptal domain expert.
2
Choose your talent
Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.
3
Start your risk-free talent trial
Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.
Top talent is in high demand.
Start hiring